pycs.sim package

Submodules

pycs.sim.draw module

This module generates fake curves by “drawing” them from common sourcesplines, adding noise, tweaking microlensing.

pycs.sim.draw.sample(lc, spline)

You give me a lightcurve, and I will set the lc.mags according to the spline. I DO modify the lc object in place ! I do NOT modify the spline, of course.

If your lightcurve has mircolensing, magshifts, or fluxshift, and of course also time shifts, I will set its mags so that it matches the spline with all of these applied !

If you think about it, this is WONDERFUL !!! To do this, I have to do something like a reverse getmags().

Note that I do not add any random noise here, this is sampling only.

pycs.sim.draw.saveresiduals(lcs, spline)

You give me some optimized lcs, and the fitting spline. This function saves the residuals of the optimized lightcurves (as an attribute), in terms of non-fluxshifted magnitudes. This means that you will have to ADD these residuals (in terms of magnitude) before any fluxshifts to the spline mags in order to recover the lc. This is made so that you can reuse this particular shotnoise once fake curves are drawn.

We have to keep this function separate from draw(), as you might want to shift the curves after saving the residuals…

Warning

Call this function if your spline matches to the lcs !

pycs.sim.draw.transfershifts(lcs, reflcs, transferml=True)

I will put (copies) of all the shifts and ML from the reflcs onto the lcs. I will check that you didn’t mix up any curves.

pycs.sim.draw.draw(lcs, spline, shotnoise=None, shotnoisefrac=1.0, tweakml=None, scaletweakresi=True, tweakspl=None, keepshifts=True, keeptweakedml=False, keeporiginalml=True, trace=False, tracedir='draw', inprint_fake_shifts=None)

Wrapper to produce one set of fake lightcurves that are similar to your lcs. Give me some lightcurves, that are optimized in fluxshift, magshift, microlensing, timeshift, and the resulting spline fit representing the intrinsic variations. I will tweak the source spline and the microlensing etc, and return to you a list of fake lightcurves. These lightcurves are just blank data points, “as observed”. I will build them from scratch. But note that by default the drawn light curve will be equiped with the same shifts and ML as your lcs.

Note

I do NOT modify lcs or spline !

Note

It is perfectly ok to shift your input curves in time after the optimization, to make me build fake lightcurves with other delays ! So if you want me to draw curves that look different from yours, it is perfectly ok to give me lcs that do not match to the spline ! Same logic applies for all other shifts and microlensing.

Parameters:

• lcs – reference lightcurves to “immitate”. I will use their epochs, their time/mag/flux shifts, their microlensing, their errorbars.
• spline – reference spline from which I will draw my magnitudes.
• shotnoise –

  Select among [None, “magerrs”, “res”, “mcres”, “sigma”]

  It tells what kind of noise to add to the fake mags. This noise will, as required, be added to the “observed” data (i.e. not fluxshifted).

• shotnoisefrac – a multiplier of the shotnoise added to the curve. Set to 0.5 and I’ll add only “half” of the noise …
• tweakml – either a function, or a list of functions, that takes a list of lightcurves and tweaks their ML in place. I will use this tweaked ML to draw them. If you give a single function, I will apply it to all the curves If you give a list of functions, I will apply them to the respective curves of your lcs (i.e., give the functions in the order corresponding to your lcs !).
• scaletweakresi – scales the “residuals” obtained by tweakml according to saved residuals
• tweakspl – a function that takes a spline and returns a tweaked spline. I will use this on the sourcespline you pass me, before drawing from it.
• keepshifts – by default I will set the time/flux/mag/ shifts from your lcs also to the fake curves.
• keeptweakedml – if keepshifts is True, and keeptweakedml is True, I will keep the tweaked ML, not the input ML, on the output curves. It makes no sens to keep the ML if not keeping the shift
• keeporiginalml – if keepshifts is True, and keeporiginalml is True, I will keep the the input ML, on the output curves.
• inprint_fake_shifts – give an array of shifts corresponding to your lcs that you want to inprint in the mock curves.

Note

I will tweak the ML only of those curves that have spline ML. You probably want me to tweak the ML of all your curves ! So be sure to add some spline ML to all your curves before calling me !

pycs.sim.draw.multidraw(lcs, spline=None, optfctnots=None, onlycopy=False, n=20, npkl=5, simset='draw', simdir=None, shotnoise=None, shotnoisefrac=1.0, truetsr=8.0, tweakml=None, scaletweakresi=True, tweakspl=None, shuffle=True, verbose=True, trace=False, destpath='./')

Even higher wrapper to produce mock + tweaked lightcurves, and save them into a directory (as pickle files), in preparation for analysing them with pycs.sim.run.multirun()

The curves I return are “like observed” : they have no shifts, no ML. Just datapoints.

Parameters:

• lcs (list) – The starting-point lightcurves from which I will draw the mock data. They must match (!), i.e., have appropriate microlensing and be optimized somehow.
• spline (spline) – The source spline used to draw the new curves. Is not used (-> None) if onlycopy=True, or if optfct is specified.
• optfctnots (function) – A function to fit a spline and the ML + all shifts execpt for timeshifts. It is called after setting the true time shifts. Put at None if you want to use always the same current ML and spline.
• onlycopy – If True, I will simply save copies of the input lcs, not drawing anything.
• n – number of lightcurve-lists to simulate per pickle
• npkl – number of pickle files
• simset – give a name to your simulation !
• simdir – where should I put these simulations ?
• shotnoise (string) – Select among None, “magerrs”, “mcres”, “res”. See definitions in pycs.sim.draw.draw().
• truetsr – radius of exploration for unifomly slectrec true time shifts (passed to multidraw) Not used if draw = False, of course.
• shuffle – Shuffle the curves before feeding them into optfctnots ? If you use this, be sure to pycs.gen.lc.objsort the curves before !

Todo

Possibility to add shot noise even with draw = False. Like in the good old days. -> Separate addshotnoise from multidraw ?

Note

I will tweak the ML only of those curves that have spline ML. You probably want me to tweak the ML of all your curves ! So be sure to add some spline ML to all your curves before calling me !

pycs.sim.draw.shareflux(lc1, lc2, frac=0.01)

I add “noise” to lc1 and lc2 by randomly sharing flux between the two sources.

Parameters:frac – The stddev of the gaussian “noise” in flux, with respect to the minimum flux in the curves.

pycs.sim.plot module

Subpackage with functions to plot all kind of results from runs.

pycs.sim.plot.mad(xs)

Return the median absolute deviation. Write it myself here instead of importing it from astropy, since it will add another depenency. Work with 1d array only

@todo: for PyCS 3, will use astropy as a default module (good) and use their functions

Parameters:xs – list of values Returns:median absolute deviation

class pycs.sim.plot.delaycontainer(data, name='No name', objects=None, plotcolour='black', marker=None)

Stores the delay or error bar measurement(s) (one for each curve pair). This object is usually produced by the plot-functions hists or meanvstrue below.

markers : [ 7 | 4 | 5 | 6 | ‘o’ | ‘D’ | ‘h’ | ‘H’ | ‘_’ | ‘’ | ‘None’ | ‘ ‘ | None | ‘8’ | ‘p’ | ‘,’ | ‘+’ | ‘.’ | ‘s’ | ‘*’ | ‘d’ | 3 | 0 | 1 | 2 | ‘1’ | ‘3’ | ‘4’ | ‘2’ | ‘v’ | ‘<’ | ‘>’ | ‘^’ | ‘|’ | ‘x’ | ‘$…$’ | tuple | Nx2 array ]

__init__(data, name='No name', objects=None, plotcolour='black', marker=None)

self.data is a list of dicts, the fields depend on if it’s delays or errorbars

• delays : label, mean, med, std
• errorbars : label, tot, sys, ran, bias

__module__ = 'pycs.sim.plot'

pycs.sim.plot.newdelayplot(plotlist, rplot=7.0, displaytext=True, hidedetails=False, showbias=True, showran=True, showerr=True, showlegend=True, text=None, figsize=(10, 6), left=0.06, right=0.97, top=0.99, bottom=0.08, wspace=0.15, hspace=0.3, txtstep=0.04, majorticksstep=2, filename=None, refshifts=None, refdelays=None, legendfromrefdelays=False, hatches=None, centershifts=None, ymin=0.2, hlines=None, tweakeddisplay=False, blindness=False, horizontaldisplay=False, showxlabelhd=True)

Plots delay measurements from different methods, telescopes, sub-curves, etc in one single plot. For this I use only delaycontainer objects, i.e. I don’t do any “computation” myself.

Parameters:plotlist – Give me a list of tuples (delays, errorbars), where delays and errorbars are delaycontainer objects as written into pkl files by hists and meanvstrue.

NEW : plotlist delaycont can handle asymmetric errors (e.g. to seamlessly compare pycs with other papers’ results). Instead of the “tot” key, new “plus” and “minus” keys are used. :type plotlist: list

Parameters:

• rplot – radius of delay axis, in days.
• displaytext (boolean) – Show labels with technique names and values of delays
• hidedetails (boolean) – Do not show (ran, sys) in labels
• refshifts (list) – This is a list of dicts like {“colour”:”gray”, “shifts”:(0, 0, 0, 90)}. Will be plotted as dashed vertical lines.
• refdelays (list) – a list of tuples (delays, errorbars) to be plotted as shaded vertical zones.
• legendfromrefdelays (boolean) – if you want to display the refdelays name in the legend panel
• hatches (list) – list of hatch keyword for the refdelays plotting
• showbias (boolean) – draws a little cross at the position of the delay “corrected” for the bias.
• showran (boolean) – draws “minor” error bar ticks using the random error only.
• text (list) – Text that you want to display, in the form : [line1, line2, line3 …] where line_i is (x, y, text, kwargs) where kwargs is e.g. {“fontsize”:18} and x and y are relative positions (from 0 to 1).
• blindness (boolean) – Shift the measurements by their mean, so the displayed value are centered around 0
• horizontaldisplay (boolean) – display the delay panels on a single line. Works only for three-delay containers.
• showxlabelhd (boolean) – display or not the x label when horizontal display is True

Warning

Altough the code says I’m plotting mean and std for the measured delays, I might be using median and mad instead! This depends on how hists was called! Be careful with this…

pycs.sim.plot.newdelayplot2(plotlist, rplot=7.0, displaytext=True, hidedetails=False, showbias=True, showran=True, showlegend=True, text=None, figsize=(10, 6), left=0.06, right=0.97, top=0.99, bottom=0.08, wspace=0.15, hspace=0.3, txtstep=0.04, majorticksstep=2, filename='screen', refshifts=None, refdelays=None, legendfromrefdelays=False, hatches=None, centershifts=None, ymin=0.2, hlines=None)

Plots delay measurements from different methods, telescopes, sub-curves, etc in one single plot. For this I use only delaycontainer objects, i.e. I don’t do any “computation” myself.

Difference from newdelayplot is that the previously hatched/shaded regions are plotted as smaller points, without infos on the time-delay

Parameters:

• plotlist (list) – Give me a list of tuples (delays, errorbars), where delays and errorbars are delaycontainer objects as written into pkl files by hists and meanvstrue.
• rplot – radius of delay axis, in days.
• displaytext (boolean) – Show labels with technique names and values of delays
• hidedetails (boolean) – Do not show (ran, sys) in labels
• refshifts (list) – This is a list of dicts like {“colour”:”gray”, “shifts”:(0, 0, 0, 90)}. Will be plotted as dashed vertical lines.
• refdelays (list) – a list of tuples (delays, errorbars) to be plotted as shaded vertical zones.
• legendfromrefdelays – if you want to display the refdelays name in the legend panel
• hatches – list of hatch keyword for the refdelays plotting
• showbias (boolean) – draws a little cross at the position of the delay “corrected” for the bias.
• showran (boolean) – draws “minor” error bar ticks using the random error only.
• text (list) – Text that you want to display, in the form : [line1, line2, line3 …] where line_i is (x, y, text, kwargs) where kwargs is e.g. {“fontsize”:18} and x and y are relative positions (from 0 to 1).

pycs.sim.plot.normal(x, mu, sigma)

Plain normal distribution. You can directly apply me on numpy arrays x, mu, sigma.

pycs.sim.plot.hists(rrlist, r=10.0, nbins=100, showqs=True, showallqs=False, qsrange=None, title=None, xtitle=0.5, ytitle=0.95, titlesize=18, niceplot=False, displaytext=True, figsize=(16, 9), left=0.06, right=0.95, bottom=0.065, top=0.95, wspace=0.2, hspace=0.2, txtstep=0.04, majorticksstep=2, hideyaxis=True, trueshifts=None, filename=None, dataout=False, blindness=False, usemedian=False, outdir='./')

Comparing the delay distributions from different run result objects.

Parameters:

• rrlist – a list of runresults object.
• r – a range radius for the hists
• showqs – If True, I overplot the qs as scatter points.
• dataout – True means that I’ll write the pkl file needed to make the delayplot.
• removeoutliers – True means I remove estimates that are the farthest from the median. Use this with CAUTION !!!
• usemedian – if True, use the median and median absolute deviation instead of mean and std.

Warning

To avoid rewriting newdelayplot, if usemedian is True then I write the median and mad in the mean and std fields of the pickles. This is dangerous (and a bit stupid and lazy), but since hists() and newdelayplot() are usually called one after the other it should not create too much confusion.

Note

Actually, using median and mad as default estimators might be smarter…? To meditate for PyCS 3.0…

pycs.sim.plot.newcovplot(rrlist, r=6, rerr=3, nbins=10, nbins2d=3, binclip=True, binclipr=10.0, figsize=(13, 13), left=0.06, right=0.97, top=0.97, bottom=0.04, wspace=0.3, hspace=0.3, method='indepbin', minsamples=10, showplots=True, printdetails=True, printcovmat=True, detailplots=False, filepath=None, verbose=True)

pycs.sim.plot.measvstrue(rrlist, r=10.0, nbins=10, plotpoints=True, alphapoints=1.0, plotrods=True, alpharods=0.2, ploterrorbars=True, sidebyside=True, errorrange=None, binclip=False, binclipr=10.0, title=None, xtitle=0.75, ytitle=0.95, titlesize=30, figsize=(10, 6), left=0.06, right=0.97, top=0.99, bottom=0.08, wspace=0.15, hspace=0.3, txtstep=0.04, majorticksstep=2, displayn=True, filename=None, dataout=False, tweakeddisplay=False, blindness=False, outdir='./')

Plots measured delays versus true delays

Parameters:

• r – radius of simulation input delays to plot (x axis range)
• nbins – number of bins for the bar plot within this range.
• plotpoints – should I plot the points (scatter plot) ?
• plotrods – should I plot the avg within each bin ?
• ploterrorbars – should I add errorbars upon the bar plot ?
• sidebyside – should I plot bars side by side, or overplot them ?
• errorrange – radius of measurement errors to plot (y axis range). You can also give a tuple (low, high), to make asymetric plots.
• binclip – should I clip errors larger than binclipr days (catastrophic failures of methods) ?
• binclipr – see binclip …

pycs.sim.plot.covplot(rrlist, showpoints=False, showcontour=True, showdensity=False, fractionalresiduals=False, bins=50, smoothing=0.0, figsize=(12, 12), left=0.02, right=0.98, bottom=0.02, top=0.98, wspace=0.05, hspace=0.05, r=5.0, title=None, txtstep=0.04, filename=None)

Covariance scatter of all measurement errors. Give me a single runresults object (from a sim, with known true delays).

pycs.sim.run module

Functions to run curve shifting techniques on lightcurves produced by sim.multidraw. We define the class runresults that “holds” results obtained by curve shifting techniques (saved into pickles).

pycs.sim.run.applyopt(optfct, lcslist, **kwargs)

Applies optfct (an optimizing function that takes a list of lightcurves as single argument) to all the elements (list of lightcurves) of lcset (a list of lists of lightcurves).

Optimizes the lightcuves themselves, in place, and returns a list of the ouputs of the optimizers, corresponding to the lcslist. For instance, if the optfct output is a spline, it also contains the final r2s, that I will later save into the pkls !

About multi cpu : First try using the multiprocessing module -> failed, as the optfct cannot be pickled. Perhaps need to rethink the strategy. Second try using good old forkmap… it works !

ncpu : None = I will use all CPUs, -1 = I will use all - 1 CPUs, and otherwise I will use ncpu CPUs.

class pycs.sim.run.runresults(lcslist, qs=None, name='None', plotcolour='#008800', success_dic=None)

Summarizes the huge list of list of lightcurves as a numpy array of timeshifts and some further info, to serve as input for plots, and actual time delay determinations. This replaces the old “boot.pkl” files … The TRUE shifts are also saved (if available)

All this is not related to a particular optimization technique. Please also provide the success_dic to remove the curves where the optimiser failed.

Note the mask funcitonallity.

__init__(lcslist, qs=None, name='None', plotcolour='#008800', success_dic=None)

lcslist may or may not have “truetimeshifts”. If not, I will put 0.0 as true shifts.

qs should be a numpy array (as long as lcslist) that contains some chi2, r2, or d2 stuff to quantify how good the fit was.

All the lcs in lcslist should be in the same order (I will check this, even if slow). I will not sort these lcs (as you might want them in an unsorted order).

__len__()

The number of runs

nimages()

The number of images (4 for a quad, 2 for a double) …

__str__()

copy()

check()

applymask(mask)

Removes some of the runresults according to your mask.

gettruets()

Returns some summary stats about the true delays. Used to find histogram ranges for plots, etc.

getts()

A bit similar to gettruets, we return the median of the measured ts… Used for plots etc, not for calculations.

get_delays_from_ts()

Return the time delays, from the timeshifts. I do not account for the true timeshift. :return: dictionary containing the median, max, and min delays + delay labels

__module__ = 'pycs.sim.run'

pycs.sim.run.joinresults(rrlist)

Give me a list of runresults objects, I join those into a single one an return the latter.

pycs.sim.run.collect(directory='./test', plotcolour='#008800', name=None)

Collects the runresult objects from a directory (typically from a multirun run), and returns the joined runresults.

pycs.sim.run.multirun(simset, lcs, optfct, kwargs_optim=None, optset='multirun', tsrand=10.0, analyse=True, shuffle=True, keepopt=False, trace=False, verbose=True, destpath='./')

Top level wrapper to get delay “histograms” : I will apply the optfct to optimize the shifts between curves that you got from pycs.sim.draw.multidraw(), and save the results in form of runresult pickles.

It is perfectly ok to launch several instances of myself on the same simset, to go faster. I will process every pkl of the simset only once, and prevent other instances from processing the same files.

You can use me for a lot of different tasks. (note from VB : not to make coffee apparently)

Parameters:

• simset – The name of the simulations to run on. Those are in a directory called sims_name.
• lcs – Lightcurves that define the initial shifts and microlensings you want to use. I will take the lightcurves from the simset, and put these shifts and ML on them.
• kwargs_optim – kwargs to be passed to your optfct
• optset – A new name for the optimisation.
• optfct (function) – The optimizing function that takes lcs as single argument, fully optimizes the curves, and returns a spline, or a d2 value. Can be None if argument analyse is False (used for tests).
• tsrand – I will randomly shift the simulated curves before running the optfct This randomizes the initial conditions. (uniform distrib from -tsrand to tsrand)
• shuffle – if True, I will shuffle the curves before running optc on them, and then sort them immediatly afterwards.
• keepopt – a bit similar to Trace, but simpler : we write the optimized lightcurves as well as the output of the optimizers into one pickle file per input pickle file. {“optfctoutlist”:optfctouts, “optlcslist”:simlcslist}

pycs.sim.run.clean_simlist(simlcslist, success_dic)

pycs.sim.src module

Stuff to represent a light curve as points on a regular grid, and power spectrum plots. This allows e.g. to tweak its power spectrum by adding “correlated noise”, resample it, etc.

class pycs.sim.src.Source(spline=None, name='Source', range=(0, 10000), sampling=0.2)

A class representing a “source” lightcurve, i.e. any artificial signal whose powerspectrum we can modify, and still evaluate the curve at given irregular jds.

To do this, we work with “internal regular arrays” (and do FFTs and so on on these), and then interpolate these fine arrays to get back values for given jds.

We store the data in magnitudes.

Ideas : use the mask carried around by self.inispline.datapoints to define regions that really matter for the power specturm and stuff.

__init__(spline=None, name='Source', range=(0, 10000), sampling=0.2)

range are in jds, only used if you don’t give me a sourcespline. sampling is in days.

I will store the spline as inispline, but I will not modify it, so no need to pass me a copy.

setup()

Sets the internal regular arrays by sampling the spline. We update self.sampling, so to get an even number of points (for FFTs).

__str__()

copy()

setmean(mean=-12.0)

Shifts the magnitudes so that their mean value are at the specified level. Don’t do this, just for testing purposes ! When doing flux PS, this scales the power, that’s all.

addgn(sigma=0.1, seed=None)

Don’t do this, just for testing purposes ! There is no reason to add white noise to the source !

addrw(sigma=0.1, seed=None)

Add a random walk, also for experiement (power law param is -2)

addplaw2(beta=-3.0, sigma=0.01, flux=False, fmin=None, fmax=None, hann=False, seed=None)

Next version, better Adds noise according to a power law PSD. See Timmer & Koenig 1995

power law would be -2

if hann, we soften the window (Hann window instead of tophat).

eval(jds)

I interpolate my ijds/imags to give you an array of mags corresponding to your jds This could in principle be done using the spline object made by the function below, but this is safer and faster.

spline()

I return a new pycs.gen.spl.Spline object corresponding to the source. So this is a bit the inverse of the constructor. You can then put this spline object as ML of a lightcurve, as source spline, or whatever.

Note that my output spline has LOTs of knots… it is an interpolating spline, not a regression spline !

..note:: This spline is a priori for display purposes only. To do an interpolation, it might be safer (and faster) to use

the above linear interpolation eval() function. But making a plot, you’ll see that this spline seems well accurate.

__module__ = 'pycs.sim.src'

pycs.sim.src.sourceplot(sourcelist, filename=None, figsize=(12, 8), showlegend=True, showspline=True, marker=None)

I show you a plot of a list of Source objects.

pycs.sim.src.window_hanning(x)

return x times the hanning window of len(x)

pycs.sim.src.detrend_mean(x)

Return x minus the mean(x)

class pycs.sim.src.PS(source, flux=False)

A class representing a power spectrum of a Source object.

__init__(source, flux=False)

Constructor, simply takes a Source object and calculates the power spectrum

The Source is always expressed in magnitudes, but you might want to get a powerspectrum in terms of flux or “counts”. Put flux = True, and I will convert your magnitudes into fluxes before calculating the power spectrum.

__module__ = 'pycs.sim.src'

__str__()

copy()

calcslope(fmin=0.001, fmax=0.5)

Measures the slope of the PS, between fmin and fmax. All info about this slope is sored into the dictionnary self.slope.

Should this depend on self.flux ?? No, only the constructor depends on this ! This is just fitting the powerspectrum as given by the constructor.

pycs.sim.src.psplot(pslist, nbins=0, filename=None, figsize=(12, 8), showlegend=True)

Plots a list of PS objects. If the PS has a slope, it is plotted as well.

if nbins > 0, I bin the spectra.

add option for linear plot ?

pycs.sim.twk module

Top level functions that use the src module to tweak microlensing and source splines. These are the function to pass them to draw.draw or draw.multidraw.

pycs.sim.twk.tweakml(lcs, spline, beta=-2.0, sigma=0.05, fmin=0.002, fmax=None, psplot=False, sampling=0.1)

I tweak the SplineML of your curves by adding small scale structure. I DO modify your lcs inplace.

pycs.sim.twk.tweakspl(spline, beta=-2.5, sigma=0.03, fmin=0.03333333333333333, fmax=0.2, hann=False, psplot=False)

Give me a spline, I return a tweaked version with added small scale structure. Note that the spline I return will have a LOT of knots.

I DO NOT modify your spline, but return a new one.

pycs.sim.twk.addspl(spl1, spl2, op='add')

Give me two splines, I return the addition/substraction of the two: spl1 +/- spl2

Important: I assume that the two splines have the correct jds ! The resulting spline

Parameters:

• spl1 – first spline
• spl2 – second spline
• op – ‘add’ for addition, ‘sub’ for subtraction

Returns:

spline

Module contents

Simulating “real” light curves, started from scratch in summer 2011