Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Opening time series touchstones from Agilent/Keysight VNAs (Discussion, maybe feature request) #432

Open
cweickhmann opened this issue Mar 1, 2021 · 23 comments
Open

Opening time series touchstones from Agilent/Keysight VNAs (Discussion, maybe feature request) #432

cweickhmann opened this issue Mar 1, 2021 · 23 comments
Labels
Feature Request Wished Feature Improvements Improvements of existing feature s-parameters Scattering parameters Touchstone concerning Touchstone formats reading/writing

Comments

@cweickhmann
Copy link

Hi everyone,

I often work with time sweep measurements on Agilent/Keysight devices.
Keysight decided to save time series s-parameter data in touchstone files as well, setting the frequency field to the illegal value s.
While this is not standard compliant, I think it is a very reasonable way to save data.

However, scikit-rf complains (rightfully so):

ERROR: illegal frequency_unit [%s] s
---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
...\site-packages\skrf\network.py in __init__(self, file, name, comments, f_unit, s_def, **kwargs)
    422             try:
--> 423                 self.read(fid)
    424             except UnicodeDecodeError:  # Support for pickles created in Python2 and loaded in Python3

...\site-packages\skrf\network.py in read(self, *args, **kwargs)
   2139         from .io.general import read
-> 2140         self.copy_from(read(*args, **kwargs))
   2141 

...\site-packages\skrf\io\general.py in read(file, *args, **kwargs)
    117     try:
--> 118         obj = pickle.load(fid, *args, **kwargs)
    119     except (UnpicklingError, UnicodeDecodeError) as e:

UnpicklingError: invalid load key, '!'.

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-3-9d35e9231b22> in <module>
      1 datadir = r"./data/"
      2 
----> 3 net = rf.Network(os.path.join(datadir, "m002_timeseries_18GHz.s2p"))

...\site-packages\skrf\network.py in __init__(self, file, name, comments, f_unit, s_def, **kwargs)
    429                 filename = fid.name
    430                 fid.close()
--> 431                 self.read_touchstone(filename)
    432 
    433             if name is None and isinstance(file, str):

...\site-packages\skrf\network.py in read_touchstone(self, filename)
   1774         else:
   1775             self.z0 = complex(touchstoneFile.resistance)
-> 1776         f, self.s = touchstoneFile.get_sparameter_arrays()  # note: freq in Hz
   1777         self.frequency = Frequency.from_f(f, unit='hz')
   1778         self.frequency.unit = touchstoneFile.frequency_unit

...\site-packages\skrf\io\touchstone.py in get_sparameter_arrays(self)
    418             # this return is tricky; it handles the way touchtone lines are
    419             # in case of rank==2: order is s11,s21,s12,s22
--> 420             return (v[:,0] * self.frequency_mult,
    421                     numpy.transpose(v_complex.reshape((-1, self.rank, self.rank)),axes=(0,2,1)))
    422         else:

TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

I could imagine some optional parameter ignore_frequency_unit as a solution.

So, here's my question: Is there a fundamental issue with allowing these files to be imported without manual modifications (see example below)?

Example of a time series dataset saved in a touchstone file by a Keysight PNA-X (other Keysight devices do it in the same way).
Note the units line # s S RI R 50:

!Keysight Technologies,N5247A,USxxxxxxxx,A.10.65.08
!Date: Sunday, February 28, 2021 22:39:32
!Correction: Sdd11(Off) 
!Sdc11(Off) 
!Scd11(Off) 
!Scc11(Off) 
!Balanced Topology: BBAL
!S2P File: Measurements: Sdd11, Sdc11, Scd11, Scc11:CW Time Sweep
!CW Freq: 18000000000 Hz
# s S  RI   R 50
0 0.094199046 -0.11541145 0.06673561 0.00081001967 -0.22223893 -0.021516033 0.09213315 -0.3309488
0.00024342486849737 0.094425932 -0.11540702 0.066734418 0.00076061487 -0.22221102 -0.021587938 0.092160314 -0.33095819
@arsenovic
Copy link
Member

ignore frequency is completely reasonable, i like it.

i think before i have used their csv's to save time-domain data.
although, since skrf has time-domain, so i now just save s-parameters.

@cweickhmann
Copy link
Author

I just went through the error messages again.
I think the unit cannot technically be ignored, but should simply be set to Hz for the users who know what they're doing.
There's not standard compliant way to achieve this. Neither in Touchstone 1.1 nor in 2.0, cf:

Btw: Are there other solutions to storing timeseries s-parameter datasets that you know of?

@Vinc0110
Copy link
Collaborator

Vinc0110 commented Mar 3, 2021

Btw: Are there other solutions to storing timeseries s-parameter datasets that you know of?

A very general way of storing and transferring datasets would be using HDF5.
As you can see in the examples on that website, the basic read/write operations are not too difficult and the API is available on many languages (including Python).

@arsenovic
Copy link
Member

to provide some context, ,
Network was built out of touchstone files. this is actually very limited, because when you want to sweep over any kind other parameter you have to do some sort of hack. this project had a more general solution called hfarray , https://github.com/hftools/hftools,
i also looked at using https://xarray.pydata.org/en/stable/, but didnt have enough time/energy to re-work and test everything.

@cweickhmann
Copy link
Author

Thanks for the feedback.
On handling the data, this is good info. hftool is new to me, the others I was familiar with.
My question was towards the devices but I've checked and it appears the misused Touchstone is the only way on the UI.

As a workaround I though of using a regular expression:

import re

fname = "yourfile.s2p"
modfh = open(fname[:-4] + ".mod" + fname[-4:], "wb+")

rex = re.compile(b"^\#\s+([a-zA-Z]+)(\s+.+)")
found_ex = False
with open(fname, "rb") as fh:
    for line in fh:
        if not found_ex and rex.match(line):
            found_ex = True
            match = rex.findall(line)
            line = b"# Hz " + match[0][-1] + b"\n"
        modfh.write(line)
modfh.seek(0)
net = rf.Network(modfh)
modfh.close()

net.plot_s_db()

The idea is to use this as a context manager later. The modfh.seek(0) I thought might allow me to just use the file handle modfh with skrf.Network(modfh).

@cweickhmann
Copy link
Author

cweickhmann commented Mar 5, 2021 •
edited

Okay, didn't work before because of quirks of skrf.Network and filehandles, but now it does...

Here's a contextmanager that handles it:

import tempfile
import re
from contextlib import contextmanager

@contextmanager
def touchstone_ignore_unit(fname):
    rex = re.compile(b"^\#\s+([a-zA-Z]+)(\s+.+)")
    suffix = "." + fname.split(".")[-1]
    modfh = tempfile.NamedTemporaryFile(buffering=0, delete=False, suffix=suffix)
    
    found_ex = False
    with open(fname, "rb") as fh:
        for line in fh:
            if not found_ex and rex.match(line):
                found_ex = True
                match = rex.findall(line)
                line = b"# Hz " + match[0][-1] + b"\n"
            modfh.write(line)
    modfh.seek(0)
    try:
        yield modfh
    finally:
        modfh.close()

Use it like this:

with touchstone_ignore_unit("testfile.s2p") as fh:
    net = rf.Network(fh, name="something more informative than the tempfile name ;-)")

net.plot_s_db()

Updates:

  • Forgot to import re for the regular expressions
  • skrf.Network does not seem to handle files based on a file handle. No matter what the platform. I initially thought it was a Windows-related problem, but the approach in my previous post using only a file handle does not work on Linux either because skrf.Network wants to use the name property which seems not to be available on a file handle.

@jhillairet jhillairet added Feature Request Wished Feature Improvements Improvements of existing feature s-parameters Scattering parameters labels Mar 6, 2021
@jhillairet
Copy link
Member

jhillairet commented Mar 6, 2021 via email •
edited

@cweickhmann
Copy link
Author

No strong opinions here. I think the initial problem is really due to misuse of Touchstone.
My workaround works quite reliably. Maybe it can be added to the examples or yet better an option like ignore_units can be added to skrf.Network.

Your comment, @jhillairet, is - I think - a separate one.
A somewhat unified way to handle the different kinds of data would be really great.
HDF5 kind of lends itself to that purpose. It's fairly standardised, widely adopted, documented and portable. Hence, imho, the argument that it's binary simply does not count.
But it's less the format than a structure within an HDF5 "container" that would be needed.

@jhillairet jhillairet added the Touchstone concerning Touchstone formats reading/writing label Apr 30, 2021
@FranzForstmayr
Copy link
Collaborator

I'll need a solution for this issue in the next time and decided to fix this issue.
Is everyone fine raising an Exception, if an Time series Touchstone is detected, which suggests an different Constructor, like
rf.Network.from_timeseries(file)?
I think this is a clean solution, the constructor should calculate the S-Parameters from the impulse responses, so it's treated as a Network after initialization.

@jhillairet
Copy link
Member

OK with your proposal

@FranzForstmayr FranzForstmayr mentioned this issue Oct 27, 2021
Closed
@cweickhmann
Copy link
Author

cweickhmann commented Oct 28, 2021 via email

@FranzForstmayr
Copy link
Collaborator

Yes exactly. I just figured out that my testdata is screwed up in some way. I'll have to measure again.

@scikit-rf scikit-rf deleted a comment from sheri-x5 Apr 19, 2022
@FranzForstmayr
Copy link
Collaborator

@cweickhmann
I was pretty sure there (was?) a specification for these kind of files available (at keysight?)... Do you maybe have such a specification?

I tried to measure again, however the data seems strange, in my case only S11 was transformed into time domain, whereas the remaining 3 parameters stayed the same...Do you have an example dataset in time domain and frequency domain to compare?

@cweickhmann
Copy link
Author

cweickhmann commented Nov 4, 2022 via email

@FranzForstmayr
Copy link
Collaborator

I should search for it, but I was not aware it's more than just changing the 'Hz' entry in the definition line to 's' for seconds.

I think so too, but I was sure I found a documentation about this once, and I can not find it again.

Afaik, Keysight VNAs have an odd (and frankly slightly annoying) behaviour where they update the S-parameter matrix as they sweep. I.e. in time domain, after the first sweep (e.g. with source Port 1), the data in S11 and S21 is valid (S21 will look odd usually), and S22 and S12 are simply garbage from the sweep before (or even the last frequency sweep).

Could you maybe provide valid testdata?

@cweickhmann
Copy link
Author

Here you go. This is a valid s2p file right from a Keysight N5247A PNA. It's renamed to .txt because Github would not accept .s2p.

switching.txt

@FranzForstmayr
Copy link
Collaborator

Are you sure everything is fine with the data?

!Agilent Technologies,N5247A,US50470143,A.09.90.13
!Agilent N5247A: A.09.90.13
!Date: Saturday, January 03, 2015 07:11:32
!Correction: S11(Full 2 Port(1,2)) 
!S21(Full 2 Port(1,2)) 
!S12(Full 2 Port(1,2)) 
!S22(Full 2 Port(1,2)) 
!S2P File: Measurements: S11, S21, S12, S22:CW Time Sweep
!CW Freq: 23750000000 Hz
# s S  dB   R 50
0 -12.486843 27.334291 -3.1314292 -48.88015 -4.3183527 94.157898 -25.091829 155.28807
0.032967032967033 -12.487159 27.351484 -3.1318481 -48.891125 -4.3181267 94.159744 -25.101725 155.27542
0.065934065934066 -12.485792 27.323 -3.1313319 -48.892918 -4.3183451 94.156502 -25.100018 155.28954
0.098901098901099 -12.487652 27.329481 -3.132103 -48.891663 -4.3206248 94.158195 -25.108162 155.25157
0.13186813186813 -12.489649 27.327703 -3.1317155 -48.891304 -4.318686 94.162804 -25.098158 155.2653
0.16483516483516 -12.487955 27.32765 -3.1309357 -48.89431 -4.3187885 94.156982 -25.107195 155.24843
0.1978021978022 -12.487185 27.337252 -3.1313045 -48.901257 -4.3186226 94.162056 -25.100147 155.21162

The time step of 32.9 ms indicates a frequency of about 15Hz. A multimeter seems more appropriate than a Keysight PNA ;)

The first column after the time vector starts at about -12 to -26...Is this the impulse response written in db?

I saw a similar file a while ago, every second column was empty as the impulse response does not deliver complex values.
What's about the (phase?) columns here?

@cweickhmann
Copy link
Author

cweickhmann commented Nov 13, 2022 via email

@FranzForstmayr
Copy link
Collaborator

Well, no, and that was the point of my initial post here It's not standard compliant. But replace the lower case s in the line with the hashbang, and it'll open.

I can open the file with the branch in #545. But I get a maximum frequency of about 15Hz due to the timestep of 33ms which made me curious?

@miek
Copy link
Contributor

miek commented Oct 29, 2023

I think there's been some confusion throughout this issue over the data in these files. The Keysight touchstone files posted are for measurements taken in CW time sweep mode. This mode repeatedly measures (frequency-domain) S-parameters at a single frequency point to see how they change over time. It's not a time-domain measurement, so I don't think it makes sense to be applying an FFT to it.

@cweickhmann
Copy link
Author

I think so too. The file is indeed a time series frequency domain measurement, not a time domain measurement. I am not aware that PNA-X does time domain measurements, but maybe that's only due to that specific feature missing on my former lab's device.

The way the above example file is created is by chaining subsequent S-parameter measurements at one frequency point into one file.
Alas, this is a complex representation of a signal sampled over a given integration period. This is not any directly sampled signal!

The 15 Hz rate was deliberately selected by me because I was looking at a very slow process. It'll go faster if you want it to.

@miek
Copy link
Contributor

miek commented Oct 30, 2023

Thinking about it a bit more, maybe the solution proposed in #903 (comment) of loading the file into a NetworkSet would be appropriate for this too?

@cweickhmann
Copy link
Author

Don't know about the others, but for me this approach seems a bit clunky. The sweep does not change any parameter except time. In my specific case it would be a NetworkSet with 9111 entries of one frequency point. I don't know if performance was improved, but last time I used NetworkSet, it was pretty slow to handle this.

Imho, it does make a lot of sense for power- or other parameter-sweeps where you usually record more than one frequency point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature Request Wished Feature Improvements Improvements of existing feature s-parameters Scattering parameters Touchstone concerning Touchstone formats reading/writing
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@miek @arsenovic @jhillairet @FranzForstmayr @cweickhmann @Vinc0110