Hi there,

I'm trying pyJoules in a linux docker (ubuntu 16.04) running on MacOS 10.13.6.

However, when I try to run a simple script about pyJoules (shown as below), I get error information saying "TypeError: reduce() of empty sequence with no initial value". Do you know how to solve this issue? Thanks!

This is the script:

from pyJoules.energy_meter import measure_energy
@measure_energy
def foo(num_itr=1000000):
rst = 0
for i in range(num_itr):
rst *= i

foo(100000000)

This is the error information:

Traceback (most recent call last):
File "pyJoules_test.py", line 10, in
foo(100000000)
File "/miniconda/envs/benchmark_gnn/lib/python3.7/site-packages/pyJoules/energy_meter.py", line 298, in wrapper_measure
handler.process(energy_meter.get_trace())
File "/miniconda/envs/benchmark_gnn/lib/python3.7/site-packages/pyJoules/energy_meter.py", line 160, in get_trace
return self._generate_trace()
File "/miniconda/envs/benchmark_gnn/lib/python3.7/site-packages/pyJoules/energy_meter.py", line 169, in _generate_trace
domains = self._get_domain_list()
File "/miniconda/envs/benchmark_gnn/lib/python3.7/site-packages/pyJoules/energy_meter.py", line 166, in _get_domain_list
return reduce(operator.add, [device.get_configured_domains() for device in self.devices])
TypeError: reduce() of empty sequence with no initial value

I am trying to monitor a single process, which should be possible according to the "Getting Started" docs.

Smartwatts is made for tracking the energy consumption of processes on a machine.
[...]
If you need to monitor a process or a group of process via SmartWatts, please follow this tutorial.

I followed the tutorial, enabled cgroup v1 and created a cgroup for my process. How can I pass the cgroup to the Smartwatts formula or the HWPC sensor? I do not find the CLI argument to do that.

Thank you for your tool!

Hi,

I am trying to compare energy footprint of two different Intel processors. I am able to collect energy data for all the domains ("package_0; dram_0; core_0; uncore_0") on "Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz". However, I only see data for "dram_0;dram_1" domains for the "Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz".

I am using a decorator to monitor energy.

Any suggestions?

Thanks!

I've been using pyJoules to measure the energy consumption of a rather long machine learning inference and I have, in one case, a negative value. In the output cvs file, I have the following :

timestamp;tag;duration;package_0
1614164969.4923563;;288.83703422546387;-253371267933.0

I'm not using the decorator but instead I create an EnergyMeter manually:

    devices = DeviceFactory.create_devices([RaplPackageDomain(0)])
    pj_meter = EnergyMeter(devices)
    pj_meter.start()

    # Actual work here ...
    
    pj_meter.stop()
    trace = pj_meter.get_trace()
    handler = CSVHandler(filename=f"{args.pyjoules}/pyjoules.csv")  
    handler.process(trace)
    handler.save_data()

One thing I have noted, is that is seems to happen on a two-socket machine, but not on my laptop (although that could very well be a coincidence).

Any idea on what could happen here ? is there anything I can do to help debugging it ?

the class that will be mainly used (measureit, EnergyContext, Sample) must be located in the pyJoules package

Hello,

When I run pyJoules, I have the following error:

PermissionError: [Errno 13] Permission denied: '/sys/class/powercap/intel-rapl/intel-rapl:0/intel-rapl:0:2/energy_uj'

I do not understand what I did to have less permission on these files.

Here, is the output of ls -l:

ls -l /sys/class/powercap/intel-rapl/intel-rapl:0/intel-rapl:0:2 
total 0
-r-------- 1 root root 4096 Nov 15 15:19 constraint_0_max_power_uw
-r-------- 1 root root 4096 Nov 15 15:19 constraint_0_name
-rw------- 1 root root 4096 Nov 15 15:19 constraint_0_power_limit_uw
-rw------- 1 root root 4096 Nov 15 15:19 constraint_0_time_window_us
lrwxrwxrwx 1 root root    0 Nov 15 15:19 device -> ../../intel-rapl:0
-rw-r--r-- 1 root root 4096 Nov 15 15:19 enabled
-r-------- 1 root root 4096 Nov 15 11:10 energy_uj
-r-------- 1 root root 4096 Nov 15 15:19 max_energy_range_uj
-r--r--r-- 1 root root 4096 Nov 15 15:14 name
drwxr-xr-x 2 root root    0 Nov 15 15:19 power
lrwxrwxrwx 1 root root    0 Nov 15 15:19 subsystem -> ../../../../../../class/powercap
-rw-r--r-- 1 root root 4096 Nov 15 15:19 uevent

Looks quite normal that the permission is denied, because of -r-------- 1 root root 4096 Nov 15 11:10 energy_uj.
I do not want to set up the proper permissions manually(e.g. with chmod command), do you have any idea if there are any settings or something I can do to have the proper permissions ?

Thank you.

Is it possible to use pyjoules to profile the energy on raspberry pi? Thanks!

Hi,

How do we cite your package in our academic paper?

Do you guys have a bibtex file I could use?

@altor @rouvoy @chakib-belgaid @danglotb

I have the following code snippet running on Ubuntu 20.04 on Windows Subsystem for Linux (WSL).

from pyJoules.energy_meter import measure_energy

@measure_energy
def foo():
    print("HERE")

foo()

I get the following stack trace error:

WARNING:root:pynvml not found you can't use NVIDIA devices
HERE
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    foo()
  File "/home/kshivvy/venv/lib/python3.8/site-packages/pyJoules/energy_meter.py", line 298, in wrapper_measure
    handler.process(energy_meter.get_trace())
  File "/home/kshivvy/venv/lib/python3.8/site-packages/pyJoules/energy_meter.py", line 160, in get_trace
    return self._generate_trace()
  File "/home/kshivvy/venv/lib/python3.8/site-packages/pyJoules/energy_meter.py", line 169, in _generate_trace
    domains = self._get_domain_list()
  File "/home/kshivvy/venv/lib/python3.8/site-packages/pyJoules/energy_meter.py", line 166, in _get_domain_list
    return reduce(operator.add, [device.get_configured_domains() for device in self.devices])
TypeError: reduce() of empty sequence with no initial value

Doe pyJoules support WSL? Would I need to do a full Linux partition on my laptop or buy a machine with Linux preinstalled? Or is there an alternative way to profile energy used by specific lines of code?

Thanks,
Keshav

Hi, is it possible to measure energy of raspberry pi. I tried it but failed to measure.

It shows an error-

Traceback (most recent call last):
  File "/home/master-pi/test.py", line 7, in <module>
    main()
  File "/home/master-pi/.local/lib/python3.11/site-packages/pyJoules/energy_meter.py", line 298, in wrapper_measure
    handler.process(energy_meter.get_trace())
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master-pi/.local/lib/python3.11/site-packages/pyJoules/energy_meter.py", line 160, in get_trace
    return self._generate_trace()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master-pi/.local/lib/python3.11/site-packages/pyJoules/energy_meter.py", line 169, in _generate_trace
    domains = self._get_domain_list()
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/master-pi/.local/lib/python3.11/site-packages/pyJoules/energy_meter.py", line 166, in _get_domain_list
    return reduce(operator.add, [device.get_configured_domains() for device in self.devices])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: reduce() of empty iterable with no initial value

Hello,

I'm running pyJoules on keon/algorithms by putting decorator on all functions and calling the save of csv_handler at the end of each unit test.

Basically, I have the following:

@pyJoules.energy_meter.measureit(handler=pyjoules_handler.handler)
def quick_sort(arr, simulation=False):
    """ Quick sort
        Complexity: best O(n log(n)) avg O(n log(n)), worst O(N^2)
    """
    
    iteration = 0
    if simulation:
        print("iteration",iteration,":",*arr)
    arr, _ = quick_sort_recur(arr, 0, len(arr) - 1, iteration, simulation)
    return arr

and

def test_quick_sort(self):
        self.assertEqual([1, 5, 23, 57, 65, 1232],
                         quick_sort([1, 5, 65, 23, 57, 1232]))

        pyjoules_handler.handler.save_data()

for all functions/tests.

I have the following errors when I run the tests:

======================================================================
ERROR: test_huffman_coding (test_compression.TestHuffmanCoding)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/workspace/algorithms/tests/test_compression.py", line 27, in test_huffman_coding
    HuffmanCoding.encode_file(self.file_in_name, self.file_out_bin_name)
  File "/home/benjamin/.local/lib/python3.7/site-packages/pyJoules/energy_meter.py", line 286, in wrapper_measure
    val = func(*args, **kwargs)
  File "/home/benjamin/workspace/algorithms/algorithms/compression/huffman_coding.py", line 292, in encode_file
    with open(file_in_name, "rb") as file_in, open(file_out_name, mode="wb+") as file_out:
OSError: [Errno 24] Too many open files: 'huffman_coding_out.bin'

======================================================================
ERROR: test_huffman_coding (test_compression.TestHuffmanCoding)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/workspace/algorithms/tests/test_compression.py", line 40, in tearDown
    os.remove(self.file_out_bin_name)
FileNotFoundError: [Errno 2] No such file or directory: 'huffman_coding_out.bin'

======================================================================
ERROR: test_histogram (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_histogram
Traceback (most recent call last):
  File "/usr/lib/python3.7/unittest/loader.py", line 436, in _find_test_path
  File "/usr/lib/python3.7/unittest/loader.py", line 377, in _get_module_from_name
  File "/home/benjamin/workspace/algorithms/tests/test_histogram.py", line 1, in <module>
  File "/home/benjamin/workspace/algorithms/algorithms/distribution/histogram.py", line 20, in <module>
  File "/home/benjamin/.local/lib/python3.7/site-packages/pyJoules/energy_meter.py", line 280, in decorator_measure_energy
  File "/home/benjamin/.local/lib/python3.7/site-packages/pyJoules/energy_device/energy_device_factory.py", line 63, in create_devices
  File "/home/benjamin/.local/lib/python3.7/site-packages/pyJoules/energy_device/energy_device_factory.py", line 47, in _gen_all_available_domains
  File "/home/benjamin/.local/lib/python3.7/site-packages/pyJoules/energy_device/rapl_device.py", line 111, in available_domains
  File "/home/benjamin/.local/lib/python3.7/site-packages/pyJoules/energy_device/rapl_device.py", line 135, in available_package_domains
OSError: [Errno 24] Too many open files: '/sys/class/powercap/intel-rapl/intel-rapl:1/name'

I show the 3 first errors, then the repeated error is OSError: [Errno 24] Too many open files: '/sys/class/powercap/intel-rapl/intel-rapl:1/name'

It seems like pyJoules opens files without closing them. Do you have any clue? Is there a solution or should I just run the test one by one?

Best, thank you very much!

Hello,

I could read in the README:

Here are some basic usages of pyJoules. Please note that the reported energy consumption is not only the energy consumption of the code you are running. This includes the global energy consumption of all the process running on the machine during this period, thus including the operating system and other applications. That is why we recommend to eliminate any extra programs that may alter the energy consumption of the machine hosting experiments and to keep only the code under measurement (i.e., no extra applications, such as graphical interface, background running task...). This will give the closest measure to the real energy consumption of the measured code.

However, while browsing the source code of pyJoules, I found the function remove_idle.

Does this function remove the idle period of the process that we want to measure? Or this function does something else which is not related to the mentioned excerpt of the documentation?

Thank you very much!

Best

@altor @rouvoy @chakib-belgaid @danglotb
I have the following code snippet running on Ubuntu 22.04.1

from pyJoules.energy_meter import EnergyContext
from pyJoules.device.rapl_device import RaplDramDomain
from pyJoules.handler.csv_handler import CSVHandler


csv_handler = CSVHandler('result.csv')

with EnergyContext(handler=csv_handler, domains=[RaplDramDomain(0)], start_tag='Test') as ctx:
    train()
 csv_handler.save_data()

I get the following stack trace error:

Traceback (most recent call last):
  File "/home/hafizur/test.py", line 8, in <module>
    with EnergyContext(handler=csv_handler, domains=[RaplDramDomain(0)], start_tag='Test') as ctx:
  File "/home/hafizur/anaconda3/envs/myenv/lib/python3.10/site-packages/pyJoules/energy_meter.py", line 322, in __init__
    devices = DeviceFactory.create_devices(domains)
  File "/home/hafizur/anaconda3/envs/myenv/lib/python3.10/site-packages/pyJoules/device/device_factory.py", line 78, in create_devices
    device.configure(domains=grouped_domains[device_type])
  File "/home/hafizur/anaconda3/envs/myenv/lib/python3.10/site-packages/pyJoules/device/rapl_device.py", line 211, in configure
    Device.configure(self, domains)
  File "/home/hafizur/anaconda3/envs/myenv/lib/python3.10/site-packages/pyJoules/device/device.py", line 64, in configure
    raise NoSuchDomainError(domain)
pyJoules.exception.NoSuchDomainError

Does pyJoles able to measure the energy consumption of DRAM?
N.B: It can measure energy consumption of NVIDIA GPU. But think it measures energy in milli-joule, not micro-joule

          I faced the same issue and taking note of what nikhil153 mentioned, did the following fix in line 136 of rapl_device.py:

if domain_name_file.readline() == 'package-' + str(socket_id) + '\n': #removed this
if "package-" in domain_name_file.readline(): #replaced with this

Hope this helps others too!

Originally posted by @venugopalansgr in #14 (comment)

PowerMeter Réunitilisable

Est ce que le powerMeter doit pourvoir être utilisé plusieurs fois d'affilé ? si c'est le cas, lors d'une nouvelle utilisation, la trace précédente doit être effacé. Quand procède-t-on à cette action ?

Why is there a restriction to python >= 3.7 ? Could this be lossened?

Hello,

I'm trying to install pyJoules using pip3 but I have the following error:

$ pip3 install pyJoules
Collecting pyJoules
  Could not find a version that satisfies the requirement pyJoules (from versions: )
No matching distribution found for pyJoules

I did try pip install pyJoules, resulting in the same message.

Any idea?

Thank you very much !

Here some info about my installation:

$ pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

$ python --version
Python 2.7.17

$ pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

$ python3 --version
Python 3.6.9

some rows contains negative energy

I am monitoring energy consumption of a pytorch model. I am sampling several times during training loop with EnergyContext and record (code snippet below). I am noticing that there are more than half samples showing zero consumption. See attached partial log.
joules_sample.log

Any ideas?

for i, (images, labels) in enumerate(train_loader):
    # get the inputs; data is a list of [inputs, labels]
    images = images.to(device)
    labels = labels.to(device)
  
    # zero the parameter gradients
    optimizer.zero_grad()
  
    # Monitor joules sparingly
    if (i % monitor_interval) == (monitor_interval-1):
        if monitor_joules:
            # pyjoules
            with EnergyContext(handler=pd_handler, start_tag='forward') as ctx:
                # forward + backward + optimize
                outputs = model(images)
                ctx.record(tag='loss')
                loss = criterion(outputs, labels)
                ctx.record(tag='backward')  
                loss.backward()
                ctx.record(tag='step')
                optimizer.step()
                ctx.record(tag='overhead')

Hi,

I am trying to install pyJouls via pip. But I get the following error:

ERROR: Could not find a version that satisfies the requirement pyJouls (from versions: none)
ERROR: No matching distribution found for pyJouls
Much thankful for your kind support here

Piyumal

Adding support for measuring energy cost of small code snippets, like timeit does: https://docs.python.org/3/library/timeit.html

Hello, I did succeed to run pyJoules on small examples.

However, I'm a bit confused about the output. Would you mind explaining what are the fields, their units, and how there are measured? If there is any reference that I should read, please could provide me a pointer or something likewise?

Toy python script used:

from pyJoules.energy_meter import measureit
import sys

@measureit
def fib(n,x=[0,1]):
    for i in range(abs(n)-1): 
        x=[x[1],sum(x)]
    return x[1]*pow(-1,abs(n)-1) if n<0 else x[1] if n else 0

print(fib(int(sys.argv[1])))

Output:

begin timestamp : 1599722530834595361; tag : fib; duration : 656476; package_0 : 0.0; dram_0 : 0.0; core_0 : 6531.0; uncore_0 : 0.0

Thank you very much.

Hi all

I'm at level 0 in python :( ... I've installed piJoules with pic
Try just ton run an example:
from pyJoules import *

def foo():
print("ok");

Usage 1 (basic)

with EnergyContext:
foo()

foo()

but have a compilation error
Traceback (most recent call last):
File "/home/blabla/python/t.py", line 7, in
with EnergyContext:
NameError: name 'EnergyContext' is not defined

I use python3.10, have rapl
Any idea please?

Hi ,

How to see the output after,

from pyJoules.device import DeviceFactory
from pyJoules.device.rapl_device import RaplPackageDomain, RaplDramDomain
from pyJoules.device.nvidia_device import NvidiaGPUDomain
from pyJoules.energy_meter import EnergyMeter

domains = [RaplPackageDomain(0), RaplDramDomain(0), NvidiaGPUDomain(0)]
devices = DeviceFactory.create_devices(domains)
meter = EnergyMeter(devices)

meter.start(tag='foo')
foo()
meter.record(tag='bar')
bar()
meter.stop()

trace = meter.get_trace() ?

I mean how to print it?

@altor @rouvoy @chakib-belgaid @danglotb

I have some confusion about the outputs that pyJoules returns.

It return as follows:
begin timestamp : 1675697491.336882; tag : foo; duration : 3.7640533447265625; package_0 : 153976351.0; core_0 : 127869606.0; uncore_0 : 0.0; nvidia_gpu_0 : 192276

My question is, if I want to measure the CPU energy consumption, what values should I take into account?
What is the unit of package 0?

I think package_0 return energy in uJ (1e-6J) and Nvidia_gpu return in mJ (1e-3J)
Could you please help me clarify this issue?

My machine details:
11th Gen Intel® Core™ i9-11900 @ 2.50GHz × 16
NVIDIA GeForce RTX 3060

Hi, thanks for building this library!

I see in the docs and from the issue #6 that the energy consumption is supposed to be in uJ (1e-6J), but this doesn't seem to match for the GPU output. It seems that I am getting the output in mJ (1e-3J) instead.

With an NVidia Titan RTX machine that consumes 16Watt when idle and 280Watt when fully active, we should use for 60seconds respectively 16*60=960 Joules (idle) and 280*60=16,800 Joules (active). But PyJoules is measuring 952,494 uJ = 0.95 Joules (idle) and 16,522,288 uJ = 16.52 Joules (active).

Could you confirm the metrics used for nvidia_gpu outputs?

Thanks in advance,
Johanna