Fix docker image

[ivan-magda]

By running docker run -it --rm srcd/ml --help I was not able to run docker image with the following error messages:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 570, in _build_master
    ws.require(__requires__)
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 888, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 779, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (bblfsh 2.9.14 (/usr/local/lib/python3.4/dist-packages), Requirement.parse('bblfsh==2.9.13'), {'sourced-jgit-spark-connector'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/srcml", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 3095, in <module>
    @_call_aside
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 3079, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 3108, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 572, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 585, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 779, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (bblfsh 2.9.14 (/usr/local/lib/python3.4/dist-packages), Requirement.parse('bblfsh==2.9.13'), {'sourced-jgit-spark-connector'})

jgit-spark-connector requires bblfsh with the version of 2.9.13.

After this change received the following error message:

/usr/local/lib/python3.4/dist-packages/sourced/ml/cmd/id2vec_preprocess.py:10: UserWarning: Tensorflow is not installed, dependent functionality is unavailable.
  warnings.warn("Tensorflow is not installed, dependent functionality is unavailable.")
/usr/local/lib/python3.4/dist-packages/sourced/ml/cmd/run_swivel.py:21: UserWarning: Tensorflow is not installed, dependent functionality is unavailable.
  warnings.warn("Tensorflow is not installed, dependent functionality is unavailable.")
Traceback (most recent call last):
  File "/usr/local/bin/srcml", line 11, in <module>
    load_entry_point('sourced-ml==0.8.2', 'console_scripts', 'srcml')()
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 476, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2700, in load_entry_point
    return ep.load()
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2318, in load
    return self.resolve()
  File "/usr/local/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2324, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.4/dist-packages/sourced/ml/__main__.py", line 8, in <module>
    from sourced.ml import cmd
  File "/usr/local/lib/python3.4/dist-packages/sourced/ml/cmd/__init__.py", line 22, in <module>
    from sourced.ml.cmd.id2role_eval import id2role_eval
  File "/usr/local/lib/python3.4/dist-packages/sourced/ml/cmd/id2role_eval.py", line 6, in <module>
    import pandas
ImportError: No module named 'pandas'

Move on to the setup.py pandas should be not in extras_require, but in install_requires group and as of python 3.4 usage pandas<0.21 should be used.

With these changes, I was able to build an image and run a container.

[vmarkovtsev]

@zurk I think that setting the version to 2.9.13 will break the downstream packages which require a newer version... Thoughts?

[zurk]

The best option I see is to ask data retrieval team to update bblfsh requirement and publish the new package. The problem is sourced-jgit-spark-connector is deprecated...

[vmarkovtsev]

Yes, it is not going to be updated, also there may be a backward incompatibility. Besides, we will soon have to switch to v3. I think that we should accept this change and work around it in dependent projects.

[zurk]

agree.

[vmarkovtsev]

Thank you for testing the docker build! I am going to include it to the CI so that we never break it again.

[vmarkovtsev]

This is going to be resolved after we split this project into the library and the framework, stay tuned.