paulross / skiplist Goto Github PK
View Code? Open in Web Editor NEWA C++ skip list with Python bindings. This can be used to calculate a rolling median - fast!
License: MIT License
skiplist's People
Contributors
Stargazers
Watchers
skiplist's Issues
Python skip list of long does not perform a bounds check.
Python longs are mapped to C long long and the cSkipList module provides bounds for the implementation size with min_long() and max_long(). However PySkipList<long> does not check that values are within bounds. It should raise an OverfowError on insert() and remove().
pragma mark on gcc 11.4.0
When running pip install orderedstructs I got
In file included from src/cpp/SkipList.h:546,
from src/cpp/SkipList.cpp:15:
src/cpp/Node.h:47: error: ignoring ‘#pragma mark ’ [-Werror=unknown-pragmas]
47 | #pragma mark -
| compilation terminated due to -Wfatal-errors.
on my Ubuntu with gcc 11.4.0
Quick workaround was to
pip install orderedstructs --no-clean --verbose
cd /tmp/pip-install-2trguabw/orderedstructs_9e35897d0af3405b8af47356cb93a4e4/
vi setup.py
python setup.py bdist_wheel
python setup.py install
Where these two lines are removed from setup.py
'-Werror',
'-Wfatal-errors',
Add index(value) method that returns the index of the value
A complement to at(index) and has(value) would be an index(value) that return the index of the first occurrence of the value.
Create sizeof function that estimates the memory usage of the skip list
This would be useful in seeing the speed/size trade off of using different coin toss algorithms.
can not install from pip or with `setup.py`
I got the same error src/cpp/Node.h:21: error: ignoring ‘#pragma mark ’ [-Werror=unknown-pragmas] as in #8 when install from pip
Then I install with USING_GCC=1, but I got:
$ USING_GCC=1 python setup.py install
running install
running bdist_egg
running egg_info
writing orderedstructs.egg-info/PKG-INFO
writing dependency_links to orderedstructs.egg-info/dependency_links.txt
writing top-level names to orderedstructs.egg-info/top_level.txt
reading manifest file 'orderedstructs.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'orderedstructs.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
building 'orderedstructs' extension
/home/auderson/miniconda3/bin/x86_64-conda-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/auderson/miniconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/auderson/miniconda3/include -fPIC -I. -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1 -I/home/auderson/miniconda3/include/python3.8 -c src/cpy/cOrderedStructs.cpp -o build/temp.linux-x86_64-3.8/src/cpy/cOrderedStructs.o -Wall -Wextra -Werror -Wfatal-errors -Wno-c++11-compat-deprecated-writable-strings -std=c++11 -Isrc/cpp -USKIPLIST_THREAD_SUPPORT -Wno-stdlibcxx-not-found -Wno-unknown-pragmas -Wno-cast-function-type -Wno-missing-field-initializers -g3 -O2 -DNDEBUG
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
cc1plus: error: unrecognized command line option '-Wno-stdlibcxx-not-found' [-Werror]
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
error: command '/home/auderson/miniconda3/bin/x86_64-conda-linux-gnu-cc' failed with exit status 1gcc version
$ gcc --version
gcc (GCC) 4.8.5
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.Make SkipList thread safe in Python when comparing user defined objects
If there are user defined objects a callback is made into Python code to compare them. At this point Python could switch threads to a thread that mutates the SkipList. When Python switches back to the original thread it's search state within the SkipList will likely be invalid causing a segfault or undefined behaviour.
There are (currently skipped) tests in tests/unit/test_SkipList_PyObject_threaded.py that demonstrate this failure.
Provide producer/consumer APIs for the C++ SkipList
The C++ thread-safe SkipList could be used as producer/consumer priority queue. The producer API already exists with insert(). A consume API could be a (blocking) call to, say, pop_front() or pop_back() that simultaneously checks for a non-empty SkipList and removes and returns front/back object.
The API would be something like int pop_back(T &value). If this returns 0 there is nothing to pop and value is untouched. If this returns 1 then value is the value removed from the back. Similarly for pop_front.
cannot install python package in Linux container
I am unable to install orderedstructs in a Linux docker container. I originally tried with the python:3.9.9-bullseye image, which failed and I speculated that it was because there is no C++ 11 compiler for that environment. I then tried an image based on ubuntu:jammy which does have C++ 11 compiler available:
ARG VARIANT="jammy"
FROM ubuntu:${VARIANT}
ARG PYTHON_VERSION=3.9
# The following was actually performed in the running container
# The installation of python prompts for information and so
# doesn't work non-interactively
RUN apt update \
&& apt install -y build-essential python${PYTHON_VERSION} python3-pip \
&& apt-get autoremove -y \
&& apt-get clean -y \
&& rm -rf /var/lib/apt/lists/*
When doing the pip install I get:
...
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1 -I/usr/include/python3.9 -c src/cpp/SkipList.cpp -o build/temp.linux-x86_64-3.9/src/cpp/SkipList.o -Wall -Wextra -Werror -Wfatal-errors -Wno-c++11-compat-deprecated-writable-strings -std=c++11 -Isrc/cpp -USKIPLIST_THREAD_SUPPORT -Wno-stdlibcxx-not-found -g3 -O2 -DNDEBUG
In file included from src/cpp/SkipList.h:397,
from src/cpp/SkipList.cpp:15:
src/cpp/Node.h:21: error: ignoring ‘#pragma mark ’ [-Werror=unknown-pragmas]
21 | #pragma mark -
|
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
error: command '/usr/bin/gcc' failed with exit code 1
...
I also tried installing clang and using that as the compiler. It fails as well, but differently:
...
clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I. -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1 -I/usr/include/python3.9 -c src/cpp/SkipList.cpp -o build/temp.linux-x86_64-3.9/src/cpp/SkipList.o -Wall -Wextra -Werror -Wfatal-errors -Wno-c++11-compat-deprecated-writable-strings -std=c++11 -Isrc/cpp -USKIPLIST_THREAD_SUPPORT -Wno-stdlibcxx-not-found -g3 -O2 -DNDEBUG
In file included from src/cpp/SkipList.cpp:15:
In file included from src/cpp/SkipList.h:397:
In file included from src/cpp/Node.h:12:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:54:
In file included from /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/tuple:38:
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/utility:95:29: fatal error: no template named 'remove_cv'; did you mean '::std::remove_cv'?
typename _Up = typename remove_cv<_Tp>::type,
^
/usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/type_traits:1537:12: note: '::std::remove_cv' declared here
struct remove_cv
^
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1
...
excuted "pip install orderedstructs" failed
Demonstrate Python thread safety with non-object types
Since the non-object supported Python types (long/float/bytes) are translated into native C++ types of long/double/std::string the compare function is made in C++ code so this should be Python thread-safe.
Write some Python tests using these types to demonstrate this.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.