About: Improving Language Modelling with Noise Contrastive Estimation

Not logged in : Login

(Sponging disallowed)

Facets (new session)
Description
Metadata
Settings
- Rule:
- Inverse Functional Properties:
- "Same As":

About: Improving Language Modelling with Noise Contrastive Estimation Goto Sponge NotDistinct Permalink

An Entity of Type : bibo:AcademicArticle, within Data Space : demo.openlinksw.com associated with source document(s)

Attributes	Values
type	http://eprints.org/ontology/ConferenceItemEPrint http://eprints.org/ontology/EPrint Article Academic Article
seeAlso	HTML Summary of #65147 Improving Language Modelling with Noise Contrastive Estimation
http://eprints.org/ontology/hasAccepted	Improving Language Modelling with Noise Contrastive Estimation (PDF)
http://eprints.org/ontology/hasDocument	Improving Language Modelling with Noise Contrastive Estimation (PDF) Improving Language Modelling with Noise Contrastive Estimation (Other) Improving Language Modelling with Noise Contrastive Estimation (Other) Improving Language Modelling with Noise Contrastive Estimation (Other) Improving Language Modelling with Noise Contrastive Estimation (Other) Improving Language Modelling with Noise Contrastive Estimation (Other)
dc:hasVersion	Improving Language Modelling with Noise Contrastive Estimation (PDF)
Title	Improving Language Modelling with Noise Contrastive Estimation
described by	https://demo.openlinksw.com/about/id/entity/https/www.cs.kent.ac.uk/people/staff/akj22/materials/CO644/export_kar_RDFN3.n3 https://demo.openlinksw.com/about/id/entity/https/raw.githubusercontent.com/annajordanous/CO644Files/main/export_kar_RDFN3.n3
Date	2018-11-18
Creator	Farhana Ferdousi Liza Marek Grzes
status	peer reviewed published
Publisher	AAAI Press
abstract	Neural language models do not scale well when the vocabulary is large. Noise contrastive estimation (NCE) is a sampling-based method that allows for fast learning with large vocabularies. Although NCE has shown promising performance in neural machine translation, its full potential has not been demonstrated in the language modelling literature. A sufficient investigation of the hyperparameters in the NCE-based neural language models was clearly missing. In this paper, we showed that NCE can be a very successful approach in neural language modelling when the hyperparameters of a neural network are tuned appropriately. We introduced the `search-then-converge' learning rate schedule for NCE and designed a heuristic that specifies how to use this schedule. The impact of the other important hyperparameters, such as the dropout rate and the weight initialisation range, was also demonstrated. Using a popular benchmark, we showed that appropriate tuning of NCE in neural language models outperforms the state-of-the-art single-model methods based on the standard LSTM recurrent neural networks.
Is Part Of	https://kar.kent.ac.uk/id/repository
Subject	QA276 Mathematical statistics QA76.87 Neural computers, neural networks
list of authors	https://kar.kent.ac.uk/id/eprint/65147#authors
presented at	The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)
is topic of	https://www.cs.kent.ac.uk/people/staff/akj22/materials/CO644/export_kar_RDFN3.n3 https://raw.githubusercontent.com/annajordanous/CO644Files/main/export_kar_RDFN3.n3
is primary topic of	HTML Summary of #65147 Improving Language Modelling with Noise Contrastive Estimation

Faceted Search & Find service v1.17_git144 as of Jul 26 2024

Alternative Linked Data Documents: iSPARQL | ODE Content Formats:

RDF

ODATA

Microdata

About

OpenLink Virtuoso version 08.03.3331 as of Aug 25 2024, on Linux (x86_64-ubuntu_noble-linux-glibc2.38-64), Single-Server Edition (378 GB total memory, 49 GB memory in use)
Data on this page belongs to its respective rights holders.
Virtuoso Faceted Browser Copyright © 2009-2024 OpenLink Software