spaCy.io

spaCy is committed to rigorous evaluation under standard methodology. Two papers in 2015 confirm that:

1. spaCy is the fastest syntactic parser in the world;
2. Its accuracy is within 1% of the best available;
3. The few systems that are more accurate are 20× slower or more.

spaCy v0.84 was evaluated by researchers at Yahoo! Labs and Emory University, as part of a survey paper benchmarking the current state-of-the-art dependency parsers (Choi et al., 2015).

Discussion with the authors led to accuracy improvements in spaCy, which have been accepted for publication in EMNLP, in joint work with Macquarie University (Honnibal and Johnson, 2015).

from __future__ import unicode_literals, print_function
from spacy.en import English
nlp = English()
doc = nlp('Hello, world. Here are two sentences.')

token = doc[0]
sentence = doc.sents.next()
assert token is sentence[0]

hello_id = nlp.vocab.strings['Hello']
hello_str = nlp.vocab.strings[hello_id]
 
assert token.orth  == hello_id  == 469755
assert token.orth_ == hello_str == 'Hello'

assert token.shape_ == 'Xxxxx'
for lexeme in nlp.vocab:
    if lexeme.is_alpha:
        lexeme.shape_ = 'W'
    elif lexeme.is_digit:
        lexeme.shape_ = 'D'
    elif lexeme.is_punct:
        lexeme.shape_ = 'P'
    else:
        lexeme.shape_ = 'M'
assert token.shape_ == 'W'

from spacy.en.attrs import ORTH, LIKE_URL, IS_OOV
 
attr_ids = [ORTH, LIKE_URL, IS_OOV]
doc_array = doc.to_array(attr_ids)
assert doc_array.shape == (len(doc), len(attr_ids))
assert doc[0].orth == doc_array[0, 0]
assert doc[1].orth == doc_array[1, 0]
assert doc[0].like_url == doc_array[0, 1]
assert list(doc_array[:, 1]) == [t.like_url for t in doc]

doc = nlp("Apples and oranges are similar. Boots and hippos aren't.")
 
apples = doc[0]
oranges = doc[1]
boots = doc[6]
hippos = doc[8]
 
assert apples.similarity(oranges) > boots.similarity(hippos)

from spacy.parts_of_speech import ADV
 
def is_adverb(token):
    return token.pos == spacy.parts_of_speech.ADV
 
# These are data-specific, so no constants are provided. You have to look
# up the IDs from the StringStore.
NNS = nlp.vocab.strings['NNS']
NNPS = nlp.vocab.strings['NNPS']
def is_plural_noun(token):
    return token.tag == NNS or token.tag == NNPS
 
def print_coarse_pos(token):
    print(token.pos_)
 
def print_fine_pos(token):
    print(token.tag_)

def dependency_labels_to_root(token):
    '''Walk up the syntactic tree, collecting the arc labels.'''
    dep_labels = []
    while token.head is not token:
        dep_labels.append(token.dep)
        token = token.head
    return dep_labels

def iter_products(docs):
    for doc in docs:
        for ent in doc.ents:
            if ent.label_ == 'PRODUCT':
                yield ent
 
def word_is_in_entity(word):
    return word.ent_type != 0
 
def count_parent_verb_by_person(docs):
    counts = defaultdict(defaultdict(int))
    for doc in docs:
        for ent in doc.ents:
            if ent.label_ == 'PERSON' and ent.root.head.pos == VERB:
                counts[ent.orth_][ent.root.head.lemma_] += 1
    return counts

def put_spans_around_tokens(doc, get_classes):
    '''Given some function to compute class names, put each token in a
    span element, with the appropriate classes computed.
 
    All whitespace is preserved, outside of the spans. (Yes, I know HTML
    won't display it. But the point is no information is lost, so you can
    calculate what you need, e.g. <br /> tags, <p> tags, etc.)
    '''
    output = []
    template = '<span classes="{classes}">{word}</span>{space}'
    for token in doc:
        if token.is_space:
            output.append(token.orth_)
        else:
            output.append(
              template.format(
                classes=' '.join(get_classes(token)),
                word=token.orth_,
                space=token.whitespace_))
    string = ''.join(output)
    string = string.replace('\n', '
')
    string = string.replace('\t', '    ')
    return string

byte_string = doc.as_bytes()
open('/tmp/moby_dick.bin', 'wb').write(byte_string)
 
nlp = spacy.en.English()
for byte_string in Doc.read(open('/tmp/moby_dick.bin', 'rb')):
   doc = Doc(nlp.vocab)
   doc.from_bytes(byte_string)

• API documentation
• Tutorials
• Annotation specs

$ pip install --upgrade spacy
$ python -m spacy.en.download all

Most updates ship a new model, so you will usually have to redownload the data.

$ conda install spacy
$ python -m spacy.en.download

With Python 2.7 or Python 3, using Linux or OSX, run:

$ pip install spacy
$ python -m spacy.en.download

The download command fetches and installs about 300mb of data, for the parser model and word vectors, which it installs within the spacy.en package directory.

$ curl https://raw.githubusercontent.com/honnibal/spaCy/master/bootstrap_python_env.sh | bash && source .env/bin/activate

The other way to install the package is to clone the github repository, and build it from source. This installs an additional dependency, Cython. If you're using Python 2, I also recommend installing fabric and fabtools – this is how I build the project.

$ git clone https://github.com/honnibal/spaCy.git
$ cd spaCy
$ virtualenv .env && source .env/bin/activate
$ export PYTHONPATH=`pwd`
$ pip install -r requirements.txt
$ python setup.py build_ext --inplace
$ python -m spacy.en.download
$ pip install pytest
$ py.test tests/

Python packaging is awkward at the best of times, and it's particularly tricky with C extensions, built via Cython, requiring large data files. So, please report issues as you encounter them.

• Fix regression in parse times on very long texts. Recent versions were calculating parse features in a way that was polynomial in input length.
• Add tag SP (coarse tag SPACE) for whitespace tokens. Ensure entity recogniser does not assign entities to whitespace.
• Rename Span.head to Span.root, fix its documentation, and make it more efficient.

• If you have the data for v0.87, you don't need to redownload the data for this release.
• You can now set tag=False, parse=False or entity=False when creating the pipleine, to disable some of the models. See the documentation for details.
• Models no longer lazy-loaded.
• Warning emitted when parse=True or entity=True but model not loaded.
• Rename the tokens.Tokens class to tokens.Doc. An alias has been made to assist backwards compatibility, but you should update your code to refer to the new class name.
• Various bits of internal refactoring

• Changed weights data structure. Memory use should be reduced 30-40%. Fixed speed regressions introduced in the last few versions.
• Models should now be slightly more robust to noise in the input text, as I'm now training on data with a small amount of noise added, e.g. I randomly corrupt capitalization, swap spaces for newlines, etc. This is bringing a small benefit on out-of-domain data. I think this strategy could yield better results with a better noise-generation function. If you think you have a good way to make clean text resemble the kind of noisy input you're seeing in your domain, get in touch.

Parser now more accurate, using novel non-monotonic transition system that's currently under review.

• Parser produces richer dependency labels following the `ClearNLP scheme`_
• Training data now includes text from a variety of genres.
• Parser now uses more memory and the data is slightly larger, due to the additional labels. Impact on efficiency is minimal: entire process still takes <10ms per document.

Most users should see a substantial increase in accuracy from the new model.

• Bug fixes for parsing
• Bug fixes for named entity recognition

Preliminary support for named-entity recognition. Its accuracy is substantially behind the state-of-the-art. I'm working on improvements.

• Better sentence boundary detection, drawn from the syntactic structure.
• Lots of bug fixes.

• Improved parse navigation API
• Bug fixes to labelled parsing

• Train statistical models on running text running text

• Alpha release