AMA Geoffrey Hinton

breandan · 3 日前

Hello Dr. Hinton! Thank you so much for doing an AMA! I have a few questions, feel free to answer one or any of them:

In a previous AMA, Dr. Bradley Voytek, professor of neuroscience at UCSD, when asked about his most controversial opinion in neuroscience, citing Bullock et al., writes:

The idea that neurons are the sole computational units in the central nervous system is almost certainly incorrect, and the idea that neurons are simple, binary on/off units similar to transistors is almost completely wrong.

What is your most controversial opinion in machine learning? Are we any closer to understanding biological models of computation? Are you aware of any studies that validate deep learning in the neuroscience community?

Do you have any thoughts on Szegedy et al.'s paper, published earlier this year? What are the greatest obstacles RBM/DBNs face and can we expect to overcome them in the near future?

What have your most successful projects been so far at Google? Are there diminishing returns for data at Google scale and can we ever hope to train a recognizer to a similar degree of accuracy at home?

akkhong · 1 日前

Hi Prof Hinton, thank you for doing this AMA - you are a role model to people like me in the field of deep learning. I have a couple of questions on activation functions:

Goodfellow et al. (2013) showed that maxout activations combined with dropout can achieve impressive performance in various standard datasets. However, many recent papers that I have read still stick to ReLU activations. Why is it that maxout is not the standard go-to non-linear activation?
If I am not mistaken, piecewise linear activations such as ReLU and maxout do not suffer from the vanishing gradient problem. Your paper along with Zeiler et al. (2013) 'On Rectified Linear Units for Speech Processing' seems to suggest that unsupervised learning does not improve performance. Does this mean that, if all we care about is the test error rate, unsupervised pre-training is not useful?

My interest is in the use of Bayesian models in machine learning, and I would really value your opinion on this matter:

What are your thoughts on Bayesian non-parametrics? In his AMA, Yann LeCun said, "... I really don't have much faith in things like non-parametric Bayesian methods for, say, computer vision. I don't think that has a future." Can you agree with this?
Do you think we can ever make Bayesian neural networks work in terms of competitiveness and scalability?

Qwerty10110 · 3 日前

What are your thought on DeepMind's Neural Turing Machines? Is this a promising approach?

ShinyGerbil · 3 日前

In your opinion, which of the following ideas contain the lowest hanging fruit for improving accuracy on today's typical classification problems:

1) Better hardware and bigger machine clusters

2) Better algorithm implementations and optimizations

3) Entirely new ideas and angles of attack

Thanks!

4geh · 1 日前

I am great fan of yours.

I think you have a rare knack for finding perspectives on ideas and problems that most people would not see without help, but when I hear you explain them a get a sense of clarity. "Yes, that's the way it should be!" You also show this amazing breadth in your knowledge, from obscure Finnish puns to deep intricacies of navigating high-dimensional space. I wish I could spend an apprenticeship with you so some of this would rub off on me, but although I can be pretty good I wouldn't be able to compete with the other candidates.

But I will at least take this opportunity to ask: What are greatest influences on your thinking? Where do you find your inspirations and what are some important principles that guide you in work and in life?

wilgamesh · 3 日前

1) What frontiers and challenges do you think are the most exciting for researchers in the field of neural networks in the next ten years?

2) Recurrent neural networks seem to have had a promising start but is not as active a field as DNNs. What are your current thoughts on such representations that model internal states that seem fundamental to understanding how the brain learns?

3) Do you personally derive insight from advances in neurobiology and neuroscience, for example new discoveries of neural correlates to behavior or do you view the biology as being mostly inspirational rather than informative?

I enjoyed taking your Coursera course and hope you can provide an updated version soon.

kkastner · 3 日前

Your Coursera course on neural networks was a huge benefit to me as a follow up to Andrew Ng's introductory Machine Learning course. It was only a few years ago, but there have been a ton of interesting research areas that have cropped up in the time since you created the course. Are there any topics you would add to that course if you redid it today? Any content you would focus on less?

Training of deep RNNs has recently seemed to get much more reasonable (at least for me), thanks to RMSProp, gradient clipping, and a lot of momentum. Are you going to write a paper for RMSProp someday? Or should we just keep citing your Coursera slides? :)

ml_man · 1 日前

Thank you for the AMA, I enjoyed your lecture on "Dark Knowledge".

Classification problems seem to get much more attention than regression. Do you work on any regression problems at Google?

What should people be aware of when working on regression problems (specifically in the context of using MLP's and things which may not be made explicit in the literature)?

4geh · 1 日前

How did you get the idea for the Boltzmann machine?

4geh · 1 日前

In addition to you being amazingly successful and appreciated as a scientist, people seem to hold you in very high regard as a friend, teacher, leader and so on. What is your philosophy when it comes to dealing the people in your life?

some_random_number · 1 日前

Do you see more and more breakthroughs coming from industrial labs (e.g. Google, Facebook, etc.) rather than Universities?

goblin_got_game · 3 日前

What are your thoughts on the recent work on Deep Generative Models and Stochastic Backpropagation [refs: 1, 2, 3]? Does this seem like a step in the right direction for creating models that leverage the power of neural nets jointly with the interpretability of probabilistic models?

mazieres · 1 日前

Comprehensibility seems to have moved away from ML as algorithms such as neural networks (among others) have amazing predictive capabilities while not allowing people to "interpret" their results, ie. knowing which features explain which prediction and why. Do you see this as inevitable ? Have you seen relevant work trying to enhance interpretability of "obfuscated" machine learning system ?

4geh · 1 日前

I was browsing through your publications list a few days ago as preparation for this, and was reminded that some of it (perhaps most notably the original Boltzmann machine article) concerns constraint satisfaction. I haven't taken the time to work with the idea to understand it at depth, but from what I do understand, I get a feeling that it may be an important concept for understanding neural networks. And yet, from what I see, it seems to have been something that was discussed much in the earlier days of artificial neural networks, and not that much in current machine learning. Do you still find constraint satisfaction an important context for thinking about what neural networks do? Why?

KarushKuhnTucker · 3 日前

There seems to be a lot of cool stuff that can be done with deep networks. Do you believe they can be analyzed theoretically? Or is it something that can be engineered to work well, but is too complicated to gain a deep (pun unintended) understanding of?

Eruditass · 3 日前

What is your view on recurrent neural networks used by Schmidhuber (and DeepMind?)? On their power, applicability, and difficulties.
Is there a class of problems and functions you believe a feed forward neural network cannot learn? How about non-feed-forward? Can it do physics simulations?
What is your view on the work towards analyzing and understanding what these networks are doing and the importance of it versus application?

wolet · 1 日前

Scott Fahlman says that if there is floating point in it that's not what brain is doing. What would be your answer to that comment?

eshenxian · 1 日前

As a graduate student doing research in machine learning, what direction do you recommend for a PhD thesis? What do you believe is the future of the field in 10 to 20 years?

sprinter21 · 1 日前

Hello Dr. Hinton ! Thanks for the AMA. I want to ask, what is the most interesting / important paper for Natural Language Processing fields in your opinion with neural network or deep learning element involved inside ? Thank you very much :D

mljoe · 1 日前

What is your opinion on multi-modal neural networks (esp. text/images)? Is it something worth investigating these days?

richardabrich · 1 日前

Hi Prof. Hinton,

I'd like to thank you for the Introduction to Machine Learning course at U of T that you and Richard Zemel taught in 2011. That was my first introduction to ML, and since then I have become somewhat obsessed.

My question is in regards to the applications of machine learning algorithms today. My guess is that your departure to Google, and Yan LeCun's departure to Facebook, were fueled by the large amounts of data and computing power that these companies are able to provide, allowing you to train bigger and better models. But I feel like they leave something to be desired in their immediate applications of this technology (e.g. tagging photos in Google+ and Facebook).

Meanwhile, there are very significant problems that could be being solved today, such as detecting disease in medical images, that aren't receiving nearly the same amount of time, effort, and resources. And this isn't due to a lack of availability of data, but rather due to inertia in making that data available to researchers, an apparent lack of interest on the part of researchers, or something else.

What are your thoughts on this matter? Why aren't machine learning benchmarks composed of medical images instead of images of cats and dogs? Why isn't there more interest in applying the latest machine learning methods to achieve tangible results in medicine? How can we rectify this situation?

sufferforscience · 3 日前

Hello Prof. Hinton! A bunch of graduates students in my research group were in the audience at a recent talk you gave on the possibility of a neural implementation for back-prop. After the talk we went out for lunch together and held an enthusiastic discussion about the talk. I think we all got the gist of the talk, but all of us were also missing a few details of your proposal and couldn't quite piece together the whole story.

Are you planning to write a paper on this topic and would you be willing to share your slides with our research group (or the general public)? We're at the Redwood Center for Theoretical Neuroscience. Thanks!

trendymoniker · 3 日前

Stochastic gradient is the training method of choice for most neural net models, yet its success depends critically on precisely setting one ore more hyperparameter values such as learning rate and momentum. For addressing this problem, what do you think of

Bayesian optimization
Automatic learning rate algorithms such as "Pesky Learning Rates" and "AdaDelta"

twainus · 2 日前

Recently in 'Behind the Mic' video (https://www.youtube.com/watch?v=yxxRAHVtafI), you said: "IF the computers could understand what we're saying...We need a far more sophisticated language understanding model that understands what the sentence means.

And we're still a very long way from having that."

Can you share more about some of the types of language understanding models which offer the most hope? Also, to what extent can "lost in translation" be reduced if those language understanding models were less English-centric in syntactic structure?

Thanks for your insights.

attia42 · 1 日前

Hi Professor Hinton, Since you joined Google lately, will your research there be proprietary? I'm just worried that the research done by one of the most important researchers in the field is being closed to a specific company.

4geh · 1 日前

You have frequently shown that you have a delightful sense of humour. What sort of entertainment do you enjoy? Would you be willing to recommend some of your favourites?

timewaitsforsome · 1 日前

do you ever foresee taking on phd students again?

98ahsa9d · 3 日前

Could you comment on Michael Jordan's answer here regarding "deep learning"?

benhamner · 1 日前

What tools and methods have you found useful for investigating what deep neural networks are learning and why they are performing in certain ways?

Do you think there's much more value left in analyzing what intermediate neurons in deep neural networks are learning (both individually and in aggregate), as well as how activation patterns vary as a function of categories of input? Do you think better software tooling can facilitate this?

duguyue100 · 2 日前

Hi, Dr. Hinton. Thank you for doing this AMA session. Well, I got three questions to ask.

SGD is THE most popular approach to train all kinds of neural networks, and the drawback is also obvious, it's a localize optimization technique. Are there any alternatives or future directions you've observed? Timothy P. Lillicrap and his colleagues from Oxford just proposed one called "Random feedback weights support learning in deep neural networks". And Prof. Amari proposed "Natural Gradient Descent".
Convolutional Neural Networks are working really well in Pattern Recognition. We observed that several generalization of ConvNet have been proposed this year. In particular, Robert Gens and Pedro Domingos' deep symmetry networks seems doing really well by utilizing symmetry group for mitigating invariant problems. What's your opinion on this architecture?
Supervised learning algorithms are dominated current deep learning world. And unsupervised learning algorithms are usually the variants of supervised learning (autoencoder is a pretty good example here). In your opinion, what's the future of unsupervised learning, given more and more loosely labeled data?

adalyac · 2 日前

Hi Prof Hinton! With early stopping, it is assumed that the model learns patterns that generalise before learning those that don't.

It seems to be the case since validation loss over training is usually this nice parabola. But why does it work like that? What is the mechanism? Are there any papers about this?

The reason for why I wonder is because: what if this is true only to a certain extent? What if the way it works, in hand-wavey terms, is that "the next pattern to be learned" is the 'biggest' or 'simplest' (BorS) one. Then, as long as the next BorS one generalises, we're good. As soon as the next-BorS one does not, and merely exists in the training sample, then we get overfit, and that worsens performance. So maybe we miss out on all of the smaller generalising patterns.

There is evidence for this intuition: a bigger dataset generalises better. A bigger dataset has sample-specific patterns too, but they are smaller or more complex. So maybe dataset size improves generalisation by pushing down the model's 'threshold' for minimum size / maximum complexity of generalising pattern it can pick up.

Then I wonder, why does the network always pick up the next biggest or simplest pattern? I've looked into the maths a bit and I wonder, is it because of gradient descent? The inverse approximation formulation of grad descent for regression makes it look like you're adding ever higher order polynomials as you go along. So maybe what happens is that you don't first learn the patterns that generalise per se, but rather the simplest patterns (that can be fitted with low order polynomial)?

4geh · 1 日前

On a darker note, I suspect that your work and the work of your colleagues is of great interest also in Fort Meade, Langley, Bluffdale, etc. I really hope I am not causing offence unnecessarily here, but I find it likely that there are channels for fairly direct transfer of knowledge from companies like Google to U.S. (and possibly some other) spy agencies. Do you share my concerns about this, and is it something that people in the machine learning community around you discuss and try to deal with? I know you have taken a stance against military funding of your research, and you have my utmost confidence and respect on moral issues, but for better and worse you, and the community at large, are creating a considerable deal of power, and I think we risk doing everyone a great disservice if we don't give some though to the potential "for worse" part as well. What are your thoughts about abuses (and uses) of machine learning and machine intelligence?

fasterthanlite · 1 日前

Did you come up with the term "dark knowledge"? If so, how do you come up with such awesome names for your models?

0ttr · 3 日前

thank you--I admire your work and have been studying machine learning with ANNs and related off and on since the mid 90s.

What's your opinion of the paper Intriguing Properties of Neural Networks? Do you think using the authors' approach to find the weaknesses and then train for them will fix the problem or is that an the algorithmic equivalent of simply kicking the can down the road? Is this paper going to be one that shakes the field up a bit or just is a bump in the road?

paralax77 · 2 日前

Do you think it's possible to design a neural network in which neurons have higher complexity? Could they pass more complex data between each other and process it? Could the connections themselves operate on data? Or do you think we should continue to use relatively simple designs ( at the lowest level )?

murbard · 1 日前

It's a bit of two questions in one, though they are related:

What are your thoughts on Mallat's scattering transform?

In general, do you see deep neural nets as trainable approximations to generative models, or as an approximation to some general manifold learning algorithm that hasn't quite been nailed yet?

Or, to rephrase, do you think the future of DNN will come from a mathematical insight: "Ah, this is what we were really doing all along!", or from gradually introducing more powerful tricks and training techniques?

benhamner · 1 日前

I've been fascinated by your work on dark knowledge and how capturing the probabilities that a network assigns to incorrect class labels can be very informative (both in learning about the incorrect classes & for better training procedures for smaller networks).

Have you looked at leveraging information farther down in the network (e.g. looking at the final layer of hidden neurons & training a smaller network to target the output of the last hidden layer)? Do you think this could be a useful direction?

nkorslund · 3 日前

I'll ask the Edge Foundation question: what is something you strongly believe is true about AI or machine learning, even though you cannot prove it?

live-1960 · 3 日前

You and your group have done a lot of work in the past on gating networks with multiplicative interactions. The LSTM network with much recent successes can be viewed as a special case of the LSTM where multiplicative interactions are handle designed and some parameters are fixed (e.g. fixed to be one). What do you see the future of LSTM-like model and more generally the gating networks with multiplicative interactions?

4geh · 1 日前

I would be very interested to have your suggestions for a few items of recommended reading and watching. Not necessarily in your professional field, although that is of course very welcome.

tabacof · 3 日前

Prof. Hinton, thank you for taking the time to be with us. What do you think about the work of Numenta and Vicarious, startups that claim to do cortical-based learning?

frankster · 3 日前

Will you run your coursera course again now that you're working at google?

CireNeikual · 2 日前

Hello Dr. Hinton,

Thank you for doing an AMA.

I am currently working a lot with HTM. Do you think HTM has a future? It seems to me that especially sparse distributed representations can drastically reduce training times and forgetting (I wrote a paper on this).

I know a lot of people in deep learning really dislike HTM since it doesn't have too many results yet. To me this seems like a chicken and egg problem: If nobody wants to research HTM because no results exist, then nobody will be there to produce results. Do you see deep learning adopting sparse distributed representations and predictive coding any time soon?

Also, why do you think reinforcement learning is so underrepresented in machine learning?

allliam · 3 日前

Professor,

Do you have any ideas about how a neural network might be able to solve the binding problem? Currently proposed solutions by cognitive scientists don't seem compatible with current NNs.

Will it require a different computational unit? A different structure? A different learning algorithm?

iMakeSense · 2 日前

What do you think are the best papers or resources for an introduction into understanding and applying neural networks to problems?

adalyac · 2 日前

[about convnets - sorry if you're getting tired of them!]

Do we understand why: 1) given sufficient data, regardless of weight initialisation, (ReLU) convnets reach their best performance? Yann LeCun was asked this at a conference and the answer was that 2) "the minima are clustered within a very small narrow band of energies, so if you have a process that's going to find a minimum, it will find one that will be as good as any minimum." But I can't find any papers about this.

Do you think that 1) can be taken as meaning convnets achieve global optimisation? If so, then would it not mean there is no better point on parameter space? Therefore, either no more progress can be made, or the function space spanned by this parameter space is not big enough?

jtromans · 1 日前

Over successive stages, the ventral visual system of the primate brain appears to develop neurons that respond selectively to particular objects or faces with translation, size and view invariance. It is possible that the relative timings of individual spikes, in particular Spike-Time-Dependent Plasticity (STDP), plays a crucial role in the self-organisation of such a system.

Do you believe that the fundamental properties of STDP could play a greater role in machine learning techniques popularized today, and if so, how?

statisticallynormal · 2 日前

Why does the statistical community appear to not be paying much attention to the recent neural network developments? Although doing better on speech and image recognition benchmarks are good, how useful are these developments to people who need statistical models for other purposes? How are they deficient?

chestervonwinchester · 1 日前

From a purely functional point of view we can approximate very well certain high level recognition processes the brain is performing at least in constrained environments (i.e., controlled data sets). We throw a big ANN or CNN at a big data set with a variety of ad-hoc techniques and out pops reasonable approximation (as measured by generalization performance) to the process that defines the "true" mapping (image => label, for instance).

Certainly, these techniques are extremely successful in applications where only the final input/output mapping is important, but what progress is being made toward understanding of underlying principles of recognition in the brain?
How has the recent success of these methods affected fields that attend more faithfully to the biological processes that are thought be accomplishing similar recognition tasks?

mazieres · 1 日前

How do you see the interaction between graph analysis and machine learning ? like community detection, centrality, degree measures, etc can be good features to fit a learning system ? anything beyond that ?

Mearis · 1 日前

Do you have any heuristics you use to determine the structure of a deep net? It seems like there is a lot of rule of thumbs and black magic involved.

iamkx · 1 日前

Now that you are no longer taking students, do you have any reflections/stories/thoughts on being a mentor? Many of the students you have supervised have gone on to be important researchers (eg. Prof Lecun/Ghahramani/many professors at UofT/in industry). Do you have some common advice to graduate students?

This seems especially important now since many Professors in ML are leaving for industry :)

wolet · 1 日前

Hello Mr. Hinton,

1) What is the relationship between your team and other teams such as Fernando Pereira's group or Google Deep Mind?

2) Do you think Deep Learning will be able to address common sense reasoning?

3) Do you think RBMs can be easily extended for temporal data such as text?

4) How would someone address structural characteristics of text without supervision? How can we extend current models?

5) Do you expect any breakthroughs in Deep Learning in near feature?

Mearis · 1 日前

Are you familiar with the philosophical papers by Putnam about multiple realizability? I've always found the parallel with the idea of multiple minimas in a non-convex neural network to be fascinating.

gursev · 1 日前

Hello Dr Hilton!

Thank you for taking out time to do an AMA. How do you think machine learning can benefit in discovery, remediation and prevention of software security issues? Few example may be, Buffer Overflows, Cross Site Scripting, Denial of Service vulnerabilities?

ford_beeblebrox · 1 日前

Hi Professor Hinton !

I. What part have visualisations like Hinton Diagrams played in developing neural nets ? Ib. What neural net visualisations are you currently excited by ?

II. If a generative neural net trained on MNIST finds a new way of 'handwriting' the digit 2 that is generally aesthetically pleasing is this usefully analagous to creativity ? IIb Is creativity an important type of thinking for machines ?

P.S. Graduate of your Coursera Course. It was hard work and utterly wonderful :D

siblbombs · 1 日前

RNNs are getting pretty hot right now, for 1 dimensional problems (sentence/document understanding) do RNNs seem like a better choice than 1d Convolutional nets?

Google has some listings for ML research on their job site, do you think (in general) that a lack of formal training can be somewhat overcome in interviews by having experience working with various ML approaches and a good understanding of what should/shouldn't work, or is the traditional higher education path still the best approach to getting a job in ML?

AmusementPork · 23 時間前

Hi Dr. Hinton, thanks a lot for taking the time!

The big advancements lately seem to stem from an increased ability to make use of large amounts of labeled data. Most areas of science aren't blessed by this 'big data deluge' yet are clearly amenable to Machine Learning (i.e. problems can be formulated in terms of input-output pairs). The work you did on Deep Lambertian Networks, as well as some of Graham Taylor's work, seemed to benefit from the ability to properly encode very specific prior knowledge about the problem into the network architecture (gating units specifically), and this led to some really cool results. By your estimation, is there a future for deep networks in the small data regime? What is your gut intuition about what can and cannot be represented by generative models such as restricted Boltzmann machines?
Do you have any anecdotes or personal musings about being a hilarious individual with a great intuition, in a field dominated by people who value rigor above all things? Have you ever felt out of place next to Computer Science type people, or did you always have sufficient clout that nobody cared how many digits of Pi you had memorized?

4geh · 23 時間前

I was both heartily entertained and fascinated to hear about the real key message of the 2006 Science paper. Would you be willing to elaborate somewhat here on what it is that happens in dimensionality expansion? One thing I specifically wonder about is how, if at all, it relates to sparse distributed representations.

And speaking of sparse distributed representations. Has anyone done the experiment by now to examine whether sparsity as a regularizer draws its effect from the same principle as Dropout?

4geh · 22 時間前

How do your preferred working tools and environments look these day?

Evolutis · 15 時間前

Hello Dr. Hinton.

First off, I am a big fan!

I am currently doing my Masters in Machine Learning not very far from Toronto and am working on image feature selection using RBM. More specifically I am trying methods to force even a non-stacked RBM to pick up on lower level features that could then be used to build more complex features. The difficulties that I have personally seen come up in training the normal RBM in this manner is that each neuron will eventually copy the average of image of the dataset. My question is, do you know of any research that measures the 'temperature' (I am using this term for hidden units that have essentially learned their feature) of a neuron and in turn disable it from learning any other features.

Also in a normal convolutional rbm, they use maxpools, I am currently working on using the idea for features but do not want to use maxpools. Do you think this is an interesting idea? What recommendations do you have for me in regards to this?

And this might be selfish, but I come from a smaller university and was wondering if UofT is open to outsiders for potential seminars. I would love to take advantage of something like that.

Thank you!

n_dimensional · 3 日前

Dear Prof. Hinton,

I am about to finish my PhD in Neuroscience and I am very fascinated by the "gray area" between neuroscience and machine learning.

If you could know the answer to one question about how the brain computes information, what would that be?

Is there any experiment in Neuroscience that you would love to see performed, yet nobody is doing it?

Thanks!

keornion · 3 日前

What do you think of the new approach based on probabilistic programming languages? This approach seems to be the most direct route to AI and introspection. See recent development, where learned code is able to replicate a bunch of distributions.

quaternion · 3 日前

Hi Dr. Hinton,

Thanks for doing the AMA; you are a legend. A few questions:

(1)

Your work could be described as loosely inspired by various biological features of the brain. However, one key feature of the brain absent from most ANNs for ML is the use of a richer vocabulary than just "activation" - namely, the various neurotransmitters and neuromodulators we know are being released by the brain to drive learning and behavior. Do you believe there is something to be gained in ANNs for ML by adopting such higher dimensionalities in neuronal signalling? What are good candidates for this approach (either specific neurotransmitters, or specific problems/tradeoffs in ML) in your view?

(2)

Do you think it's fair to say that neuroscience has been more influenced by ML than the reverse; do you see that ever changing; and what kinds of work do you think would be most promising for advancing the interaction among these fields?

Thanks for your thoughts.

madisonmay · 2 日前

Hi Dr. Hinton,

I'm curious on what your thoughts are on alternatives to your back-propagation algorithm. Do you think there is a need for a fundamentally different learning algorithm to facilitate training of very deep networks, or do you think small modifications to the back-propagation algorithm will be sufficient? In a recent paper entitled How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation , Yoshua Bengio outlines his thoughts on a novel alternative to back-propagation. What are your thoughts on this approach, and which new methods do you believe hold promise as "successors" to back-propagation for finding complex structure in in large, high-dimensional datasets?

alexmlamb · 1 日前

What do you think are the most significant functional properties of the human visual system that should inspire future research in computer vision? Do you think that using something like the saccades of human vision will be beneficial for computer vision (for example by having a recurrent neural network that learns how to move a high-resolution receptive field over an image)?
Do you think that we will find an algorithm for training neural networks that is better than gradient descent / back-propagation? Is this an area that you're actively researching?
Will more of your future work in computer vision work with still images or videos? Do you think that recurrent neural networks (like LSTM/NTM) will be the major technology for deep object recognition and detection from videos or do you think that we will need to develop a completely different architecture for this task?

serge_cell · 1 日前

Thank you Dr. Hinton!

In you "Dark Knowledge" talk you said that max pooling in your opinion is just practical solution, people using it because it works and it should be replaced by something else. Can you elaborate on this? Max pooling seems pretty natural - it provide both robustness and switching between activation subsets. What can replace/improve it?

speechMachine · 3 日前

There has been much interest in training a single deep neural network (the supervised variety MLPs) on multiple tasks in what is often called multi-task learning. Here hidden layers are kept common between tasks, and the linear regression layer is allowed to specialize towards a particular task. A good application is Microsoft's Skype Machine Translation.

Why is it that networks that learn on multiple tasks do so well compared to networks trained on just a single task?
If I were to train a single network per task, assuming all tasks are related then: Could the parameters of these networks from multiple related tasks be thought to lie on a manifold?
If 2 is true, in what way could we leverage current manifold based algorithms to learn better networks?

paralax77 · 2 日前

Do you think there are any NN designs that would encode knowledge via distance between neurons ( in a 3d or higher dimensional space), as opposed to weights of connections? Maybe a combination of both?

jostmey · 2 日前

Hello Dr. Hinton. Do you feel that there is still room for improving the learning rules used to update the weights between neurons, or do you feel that this area of research is essentially a solved problem and that all the exciting stuff lies in designing new architectures where neurons are wired together in novel ways? As a follow up question, do you think that the learning rules used to train artificial neural networks serve as a reasonable model for biological ones? Take for example the learning rule used in a Boltzmann machine: It is realistic in that it is Hebbian and that it requires alternating between a wake phase (driven by data) and a sleep phase (run in the absence of data), but is unrealistic in that a retrograde signal is used to transmit activity from the post-synaptic neuron to the pre-synaptic one.

Thanks!

atveit · 1 日前

What are the most promising algorithmic directions for model compression with the purpose of speeding_up use of large deep networks e.g. for use in mobile, wearable or implantable devices?

references:
1) Dark Knowledge - http://www.iro.umontreal.ca/~bengioy/cifar/NCAP2014-summerschool/slides/geoff_hinton_dark14.pdf

2) Learning Small-Size DNN with Output-Distribution-Based Criteria http://193.6.4.39/~czap/letoltes/IS14/IS2014/PDF/AUTHOR/IS140487.PDF

3) Accurate and Compact Large Vocabulary Speech Recognition on Mobile Devices http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41176.pdf

4) Learning in Compressed Space http://www.informatik.uni-bremen.de/~afabisch/files/2013_NN_LCS.pdf

5) Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition http://research.microsoft.com/en-us/um/people/kahe/eccv14sppnet/

test3545 · 2 日前

How deep learning is applied to dialog/conversation systems? Any research going on in the field?

Will we see smarter Siri/Google Now/Cortana anytime soon?

PLEASEPOOPONMYCHEST · 2 日前

Hi Geoff, first off thanks for answering an email of mine a few years back. If you were doing anything other than computer science/machine learning/AI/etc what would you be doing?

murbard · 1 日前

It is often claimed that rectified linear units avoid the problem of vanishing gradient, but it only seems to skew the distribution of gradients in the first layers, by having a big Dirac on 0. What do you think is the key to their success?

oddim · 1 日前

Hi Dr. Hinton,

Saw your recent talk on Dark Knowledge, and reducing large ensembles to smaller ANNs. I wondered if you had any thoughts/predictions about the limits of such reductions?

What I mean is that a network with only a handful of weights surely can't handle the ImageNet problem set, but a very large network can. But we know we can reduce some large networks without losing functionality. In essence, we're compressing the knowledge, generalizing it better, which is very useful for when we want to use the learned knowledge.

How far are we today from being able to reach the optimum minimization for a given problem? Can we even make estimates on what that limit is? Do you think we'll discover new ways to further reduce classifiers by a significant amount?

Thanks!

chao_yin · 1 日前

Deep Learning do a great job on machine learning.but can we put some prior Knowledge into deep learning , just like Graphical Model or sth else? Do you know some work on this direction?

bge0 · 1 日前

Hello Dr. Hinton! We really appreciate your time & contributions to the field. A lot of us would probably not be working/researching in this field if it weren't for you!

My question to you is this:

Where do you believe we should be focussing on to solve the problem of learning async time dependencies? RNNs seem to be able to learn a fixed number of previous time steps (even with gradient clipping). Do you believe that the best solution is along the lines of keeping memory? Like LSTMs & the recent NTM's?

aggelikiL · 1 日前

Hi Prof. Hinton and thanks in advance for doing this! I'm sure it's going to be great!

A question regarding injecting (textual) semantic information during object recognition.

Although not an expert in CV, my understanding is that some of the recognition mistakes are completely off, e.g. classifying cars as carrots as you mentioned in one talk. One intuition here is that this might be due to the "hard" labels.

In NLP, there have been advances in building really powerful and accurate distributed word vectors.

Thus, the question is why don't we use, on top of the "hard" labels (THIS IS A CAR), softer labels in the form of distributed text representations (WHAT IS A CAR). This will also allow (for example) a CNN to enforce more shareness of weights across similar objects and would most help in addressing these kind of mistakes.

Thanks!

iamtrask · 1 日前

Is your Dark Knowledge work applicable to hierarchical softmax output as well?

idre · 3 日前

Hello there, thank you for doing this. From all the questions I have got on my mind, these two are possibly the most present ones right now:

1) From the methods you described and developed, what strategy (and why) do you think is most likely employed by (parts of) the brain? I am a fan of DBNs but as an experimentalist, I find it difficult to see how that could be convincingly achieved.

2) From what I have read, most of the approaches you described deal with binary data. How would you tackle the problem of (high dimensional) continuous data (in particular time series such as LFP)?

galapag0 · 3 日前

I'm curious to know your opinion on the recent paper "Intriguing properties of neural networks", in which its authors uncover 'blind spots' in some deep neural networks. Thanks for your time on reddit!

redditnemo · 3 日前

Thank you for doing an AMA.

My questions are:

Our ability to self-reflect is important for language processing and autodidactic learning. Do you think we will see more artificial neural networks that incorporate the concept of self-reflection in the near future?
Boltzmann machines and neural networks are abstract mathematical structures and were even less tractable when they were invented some time ago. How did you do research on them without the ability to test them extensively?
What is your general approach for researching new learning methods / systems? Do you have references that you're trying to model formally (top-down) or are you working yourself up a theoretical model that 'might just work' (bottom-up)?

horsey_jumpy · 3 日前

Top 5 ML or NN researchers? Or put another way whose papers provide the most incite, fascination, into ML?

I know professors sometimes consult in industry like you're doing now at Google. Have you seen any implementations or applications of ML in industry that were surprising, funny, or novel that you can tell us about?

karmicthreat · 3 日前

Hey Dr. Hinton. Without attending graduate school for 2-4 years, what are the things newcomers need to understand to become amazing at machine learning?

Mr_Smartypants · 3 日前

My aim is to discover a learning procedure that is efficient at finding complex structure in large, high-dimensional datasets and to show that this is how the brain learns to see.

An interesting two-part goal.

When you achieve the first part, finding an efficient and powerful learning algorithm, why do you think it will be the one that the brain uses, and not some other high-performance learning algorithm? Does it have to do with your method for searching for the learning procedure?

saosebastiao · 3 日前

Are there any classes of learnable functions which are fundamentally a poor fit for neural networks (in any variation)? It seems like the only problems for which NNs are the state of the art could be classified as vision/identification. As I learn about them, they sound more and more like they the closest thing to the one true algorithm that will consume all learning algorithms, yet on most tasks relevant to my field, simple Random Forests tend to do better.

I understand that neural net success depends heavily on topology and training parameter optimization, and the range of possibilities is infinite, and therefore there are a lot of possibilities that I haven't tried. But is it possibly that my problem domain is just a bad fit for NNs in general?

StannisBaratheon_AMA · 3 日前

Hello Dr Hinton, Im doing a case study in my cog sci class related to historically significant creativity and coincidently your Fast Learning Algorithm for Deep Belief Networks is a main part of the paper. Can you speak generally on the creative process for you. As in, what is your discovery/creative process? How do you think best? What do you do to "decompress"?

I sincerely appreciate your time in responding. Thank you.

captcompile · 3 日前

Hello Professor Hinton. Thank you for taking the time to do this AMA!

I remember in one of your talks online, I think you mentioned something along the lines of, the next breakthroughs of computer vision will involve training a network to perform inverse geometry to understand the geometric structure of a scene. Is this thinking related to the work on transforming autoencoders? See this

What are your thoughts on Roland Memisevic's work on finding relations between images? See this

lars_ · 3 日前

There seem to be two approaches to understanding how brains work:

1) Work really hard on models that work, and hope to end up with insights about the brain.

2) Look really hard at actual brains, try to replicate that in models, and hope to end up with something that works.

What is the relative value of these two approaches in your opinion?

Luonnon · 3 日前

This is a less technical and more broad question for a slight change of pace! Machine learning focuses on building abstract mathematical models out of data, but these models lack any fundamental understanding of the data. For example:

A deep NN may very well be able to identify a dog in a picture X% of the time, but it does not understand the concept of a "dog."
IBM's Watson may be able to win Jeopardy by correlating words in a gigantic corpus, but the machine missed a few questions due to a lack of understanding of language.

Do you personally think that the current path of machine learning research will eventually lead to strong AI or is that breakthrough more likely to come from another field of AI?

adalyac · 2 日前

We're told sparsity is great and distributed representations are great and dropout is great.

But don't sparsity and dropout reduce the extent of distributiveness? In "Preventing co-adaptation of feature vectors", "a hidden unit cannot rely on other hidden units being present". But isn't that exactly the point of a distributed representation?

simonhughes22 · 2 日前

Dr. Hinton, long time listener, first time caller.

It seems that you can get some really impressive results from applying deep learning to the right problems. One issue I've struggled with applying DL in my own research, is that there are a lot of tricks of the trade that you need to know to get the algorithms to converge or learn at an optimal rate. Any tips or resources as to how best to acquire this knowledge?

memming · 1 日前

Hi Dr. Hinton,

How strongly does the analogy hold between cortical computations and deep networks? Can we (meta)learn about the unsupervised/reinforcement learning that presumably happens in cortex by implementing biological constraints on deep neural networks?

murbard · 1 日前

Do you think there's a future for (asymptotically) exact Bayesian methods, or do you feel that the difficulty of sampling will always give gradient descent to a MAP and tricks like dropout the upper hand?

theophrastzunz · 1 日前

Prof. Hinton,

What machine learning journals do you read regularly? Which ones would you recommend for a beginner?

feedtheaimbot · 3 日前

We've seen a tremendous improvement in NLP, vision and other fields in the past decade. What do you think will cause the next breakthrough similar to what we had with NNs? Would it be a single thing such as a new training algorithm? Or several techniques? (similar to dropout etc.)

What are your thoughts on the paper that Google Deepmind published recently? This one: http://arxiv.org/abs/1410.5401

Thoughts on HTMs?

andimai · 2 日前

Dear Prof. Hinton, what are your thoughts about the paper by Szegedy et. al.: "Intriguing properties of neural networks": - http://cs.nyu.edu/~zaremba/docs/understanding.pdf - http://www.kdnuggets.com/2014/06/deep-learning-deep-flaws.html

Did they really discover deep flaws in deep neural networks?

terath · 3 日前

If your goal is to show that the algorithms that you work with are how the brain learns, are you equally learned in neuroscience? As I understand it there are huge differences between the algorithms you work with and how the brain works. The most obvious is that the brain uses spiking neurons.

That said, I am no expert in neurology. I've always approach machine learning from a more fundamental level than mimicking biology. Very interested in your view on this topic.

lars_ · 3 日前

Are there any fields in which machine learning is not currently being applied, but where you suspect it could have a great impact?

ipu0014 · 3 日前

Having yourself used ideas from statistical physics in your work, do you think it's still worth to explore the interface between this field and machine learning in general? What would you expect the contributions to be?

stiffitydoodah · 3 日前

What's your current best guess as to how to represent/learn complex structures with a network?

Estarabim · 3 日前

Is it your view that ANNs are superior to other extant machine learning methods to solve both unsupervised and supervised learning problems? Are there certain classes of problems that ANNs are better/worse at solving than other attempted algorithmic approaches?

jtkme · 3 日前

What is your prior for the logistic unit, or similar, being the core component of state of the art deep nets 50 years from now?

watersign · 3 日前

How do you feel about tools like Weka and SPSS modeler being made available to ignorant masses such as myself? Does it ever concern you that many of these models are being used by people who have no idea how they work?

Zulban · 3 日前

Uh oh, 4 hours and no responses.

MachineLearning

調停者

ユーザインターフェイス用言語	(*) 未完成翻訳ボランティアに立候補する

MachineLearning

調停者

この操作にはログインまたは登録が必要です

新しいアカウントを作る

privacy philosophy

ログイン