pycs.tdc package¶

Submodules¶

pycs.tdc.combiconf module¶

pycs.tdc.combiconf.combiconf1(estimates)¶

Give me a group of estimates of the same TDC pair, I return the combined confidence. 1 is for doubtless 2 for plausible 3 for multimodal 4 for uninformative

Simple version

@param estimates: list of Estimate objects

@return: dictionary that contains the combined confidence code and the estimate set, rung and pair.

pycs.tdc.combiconf.combiconf2(estimates)¶

Give me a group of estimates of the same TDC pair, I return the combined confidence. 1 is for doubtless 2 for plausible 3 for multimodal 4 for uninformative

Updated version of combiconf1, now with accordance criterias between the estimates, and weight given to Vivien/Malte estimates

@param estimates: list of Estimate objects

@return: dictionary that contains the combined confidence code and the estimate set, rung and pair.

pycs.tdc.combiconf.reroll(estimates)¶

Give me a list of estimates. I remove all estimates not from Malte or Vivien, then recompute the combiconfcode Return the new combiconfcode, along with the id of the estimate

@param estimates: list of Estimate objects

@return: dictionnary containing the new combiconfcode computed by combiconf2, the estimate set, rung and pair and the corresponding estimates from Malte and Vivien only

pycs.tdc.combiconf.combiquads(estimates)¶

This function is based on a exploit discovered during the rungs generation: the quads with the same pair indices are the same amongst all rungs, thus have the same delay.

Not used for TDC1 official submissions, just for fun.

Give me a list of already combined estimates (i.e. one per pair only). I compute for each quad pair the best td and tderr among the corresponding rungs (which is an exploit !) I return the modified estimates list, with all the quads modified, and with only the doubtless double

# BIG WARNING !!! This function (pycs.tdc.combiconf.combiquads) actually modify estimates, we DO NOT want that !

@param estimates: list of Estimate objects. Each id must be present no more than once.

@return: list of modified Estimate objects.

pycs.tdc.est module¶

Stuff to manipulate time delay estimations from different techniques, specific to the TDC.

class pycs.tdc.est.Estimate(set='tdc0', rung=0, pair=1, method='None', methodpar='None', td=0.0, tderr=0.0, ms=0.0, confidence=0, timetaken=0.0)¶

Class to hold delay estimates for TDC curves, as obtained by various delay estimators.

__init__(set='tdc0', rung=0, pair=1, method='None', methodpar='None', td=0.0, tderr=0.0, ms=0.0, confidence=0, timetaken=0.0)¶

Parameters:

set – What set (e.g., “tdc0”) the curves are from
rung – rung
pair – pair
method – Name of the method
methodpar – String containing parameters of this method, or username…
td – time delay point estimate
tderr – 1sigma error estimate on the time delay
ms – magnitude shift estimate
confidence – confidence level. 0 = not estimated, 1 = doubtless, 2 = plausible, …
timetaken – seconds it took to get this estimate

__str__()¶

valstr()¶

aslist()¶

check()¶

copy()¶

getcolor()¶

setid()¶

applytolcpair(lcb)¶

__module__ = 'pycs.tdc.est'¶

pycs.tdc.est.readcsv(filepath)¶: Read a CSV file of estimates. You can specify a directory instead of a file, and I will read in all .csv files form there.

pycs.tdc.est.printestimates(estimates)¶

pycs.tdc.est.writecsv(estimates, filepath, append=False)¶: Write a CSV file of estimates. If append = True, I do NOT overwrite an existing file, but append to it !

pycs.tdc.est.importfromd3cs(filepath, set='tdc1')¶: Reads a d3cs log file and returns the list of estimates

pycs.tdc.est.select(estimates, sets=None, rungs=None, pairs=None, idlist=None)¶: Returns a sublist of the estimates selected according to the specified arguments. It can be sets/rungs/pairs values, or a list of ids

pycs.tdc.est.group(estimates, verbose=True)¶: Groups estimates by “quasar” In other words : takes a messy list of mixed estimates and returns a list of lists of estimates for a pair.

pycs.tdc.est.checkunique(estimates)¶: Checks that there is only one estimate per pair

pycs.tdc.est.sort(estimates)¶: Sorts by set, rung, pair… by id !

pycs.tdc.est.checkallsame(estimates)¶: Checks that the estimates are all about the same quasar If yes, return the set, rung and pair

pycs.tdc.est.match(candestimates, refestimates)¶

candestimates and refestimates are two lists of estimates. I return a tuple of two lists :

those estimates of candestimates which are about quasars that are present in refestimates.

the remaining candestimates, about quasars not in refestimates

pycs.tdc.est.removebad(estimates, verbose=True, crit='conflevel')¶: Remove the estimates with bad confidence level (typically 4)

pycs.tdc.est.combine(estimates, method='meanstd', methodcl=None)¶

Combine estimates according the method of your choice. Return an estimate object

list of methods (add yours): - meanstd - initialestimation - d3cscombi1 - …

Methods should define : td, tderr, confidence, ms, timetaken

pycs.tdc.est.multicombine(estimates, method='meanstd')¶: Wrapper around combine. Multiple calls to the combine function. Eats a list of messy estimates, groups and combine them by quasar according to method, and return the list of combinated estimates. Can be used to feed writesubmission.

pycs.tdc.est.show(estimates)¶: Show the estimates

pycs.tdc.est.d3cs(set, rung, pair)¶: Open d3cs with the rung/pair curve in your default browser

pycs.tdc.est.interactivebigplot(estimates, shadedestimates=None, plotpath=None, interactive=True, minibox=False, groupests=False, minradius=100)¶

WARNING !! No more TDC0-proof !!

Large graphical representation of your estimates.

Parameters:

minradius – Minimal half-width of the time delay axes, in day.
shadedestimates – Here you can give me a list of estimates that I will show as shaded bars instead of errorbars. However, it’s estimates that determines which panels I will draw and with what range, so that you can get panels shown without any shadedestimate.
plotpath – if you want to save your figures instead of displaying them. Work only when interactive is set to False.
interactive – Add a button to show the curve with shifts corresponding to the displayed estimates, and another button redirecting to d3cs
minibox –
add a minibox to the right of each panel, displaying the mean value of each set of estimate TODO : do the same implementation as shadedestimates, but with a list of different estimates

also add captions
groupests – interactive must be set to True. If groupests is True, display a command center allowing the user to navigate between the different curves selected, displaying them 10 by 10.

pycs.tdc.est.bigplot(estimates, shadedestimates=None, plotpath=None, minradius=100)¶

Large graphical representation of your estimates.

Parameters:	minradius – Minimal half-width of the time delay axes, in day. shadedestimates – Here you can give me a list of estimates that I will show as shaded bars instead of errorbars. However, it’s estimates that determines which panels I will draw and with what range, so that you can get panels shown without any shadedestimate.

pycs.tdc.metrics module¶

Functions related to the TDC metrics

pycs.tdc.metrics.fN(estimates)¶: @param estimates: list of Estimate objects @return: length of the estimates list

pycs.tdc.metrics.f(estimates, N)¶: @param estimates: list of Estimate objects @param N: maximum number of Estimates expected @return: fractional length of the estimates list with respect to N

pycs.tdc.metrics.P(estimates)¶: @param estimates: list of Estimate objects @return: Approximation of the P metric…

pycs.tdc.metrics.sortbyP(estimates)¶

I sort your estimates according to their claimed precision lowest “precision” (== highest tderr/td) first -> select from end !

@param estimates: list of Estimate objects @return: sorted list of Estimate objects.

pycs.tdc.metrics.sortbyabstd(estimates)¶

I sort your estimates according to the absolute value of their time delay. lowest “precision” (== lowest delays) first -> select from end !

@param estimates: list of Estimate objects @return: sorted list of Estimate objects.

pycs.tdc.metrics.maxPplot(estslist, N, filepath=None)¶

Give me a list of estimate-lists, I plot P as a function of f

@param estslist: list of lists of Estimate objects @param N: total number of curves (e.g. 56 for tdc0) (defines f=1) @param filepath: if not None, it defines the path where I will save the plot

pycs.tdc.metrics.getP(db, method, median=False)¶: Compute P and Perr - std(P)/sqrt(len(P)) - for a given method stored in the database

pycs.tdc.metrics.getA(db, method, median=False)¶: Compute A and Aerr - std(A)/sqrt(len(A)) - for a given method stored in the database

pycs.tdc.metrics.getAmod(db, method, median=False)¶: Compute Amod and Amoderr - std(Amod)/sqrt(len(Amod)) - for a given method stored in the database

pycs.tdc.metrics.getchi2(db, method, median=False)¶: Compute chi2 and chi2err - std(chi2)/sqrt(len(chi2)) - for a given method stored in the database

pycs.tdc.metrics.getf(db, method, N)¶: Compute f for a given method stored in the database N is the total number of curves

pycs.tdc.metrics.Pplotall(db, methods, N, zoomchi2=False, zoomA=False, errorbar=False, median=False)¶

give me the db and a list of methods [method1,method2,…], I plot P vs f for each method and chi2 vs f, A vs f… for the same arrangement according to P

If as a method_i you give me a tuple (method_i,chi2max), I will remove the values wich chi2 > chi2max from the db

I can do the same exercice with median values instead of mean, just specify it in the kwargs

pycs.tdc.metrics.Pplotallperrung(wholedb, methods, errorbar=False)¶

Ugly beast ahead…

give me the db and a list of methods [method1,method2,…], I plot P vs f for each method and chi2 vs f, A vs f… for the same arrangement according to P

If as a method_i you give me a tuple (method_i,chi2max), I will remove the values wich chi2 > chi2max from the db

pycs.tdc.metrics.combigauss(subtds, subtderrs, truetds, lensmodelsigma=0.0)¶: Give me submission delays and error bars, as well as the corresponding true delays, in form of numpy arrays. I compute the mean and sigma of the combined posterior on the fractional time delay distance error.

pycs.tdc.metrics.Pplotcombi(db, methods, N, lensmodelsigma=0.0)¶: give me the db and a list of methods, I plot combigauss params (center, sigma, zerovalue) vs f f is selected according to bestP

pycs.tdc.optfct module¶

Here we collect the optimizers specifically designed for the TDC These optimizers are to be used on lcs that have a spline already ! Typically, they are called by the runsim and runobs functions of run.py

pycs.tdc.optfct.spldiff(lcs, verbose=True, magshift=False)¶

Custom spldiff optimizer for TDC

@param lcs: list of light curves @param magshift: boolean, do you want to allow magnitude shifts in the pycs.spldiff.multiopt.opt_ts function called here

pycs.tdc.optfct.regdiff2(lcs)¶

Regdiff usign scikit-learn Let’s see if this works without any initial shift.

@param lcs: list of light curves

pycs.tdc.run module¶

Wrapper stuff to run PyCS on TDC data

We change the philosophy here: we make a clear separation between functions that draw lcs, and functions that analyse them.

The goal is to create “once and for all” a huge amount of simulated curves, and then run the optimizer on a limited number of simulated curves, randomly chosen.

The copied and simulated lcs are stored each into an individual pkl (one per lcs)

pycs.tdc.run.createdir(estimate, path)¶

pycs.tdc.run.drawcopy(estimate, path, n=1, maxrandomshift=None, datadir='')¶

Draw n times one single copy of lcs (the ones your estimate is about) into the path directory. NEW: set can be the name of a COSMOGRAIL lens.

@param estimates: list of estimate objects, with id, td and tderr @param path: where your copy will be written @param addmlfct: If you want to add different microlensing to your obslcs @param n: Number of time you will run drawobs (each run produce new copycurves) @param maxrandomshift: Maximum range for the random time shift added to the copycurves

pycs.tdc.run.drawsim(estimate, path, sploptfct, n=1, maxrandomshift=None, datadir='')¶

Draw n times one single sim curves of lcs (the ones your esimate is about) into the path directory.

@param estimates: list of estimate objects, with id, td and tderr @param path: where your copy will be written @param addmlfct: If you want to add different microlensing to your obslcs @param n: Number of time you will run drawsim (each run produce new simcurves) @param maxrandomshift: Maximum range for the random “true” and “guessed” time shift added to the simcurves

pycs.tdc.run.runcopy(estimate, path, optfct, n=1, clist=None)¶

Run the optimizer (optfct) on n different copycurves Return the optimised timeshift and magnitude for each copycurves

The n copycurves are chosen randomly in the copydir, unless you give a clist of numbers i.e. if clist = [1,3] runobs will run on c001.pkl and c003.pkl

@param estimate: give me the estimate you want me to run on. (for the id) @param path: where the copycurves are written @param optfct: which optimisation function do you want me to use @param n: on how many copies do you want me to run the optimiser @param clist: ids of specific copycurves you want me to run on.

pycs.tdc.run.runsim(estimate, path, optfct, n=1, slist=None)¶

Run the optimizer (optfct) on n different simcurves Return the optimised timeshift and magnitude for each copycurves Also return the true delays of each simcurve

The n simcurves are chosen randomly in the simdir, unless you give a slist of numbers i.e. if slist = [1,3] runobs will run on s001.pkl and s003.pkl

@param estimate: give me the estimate you want me to run on. (for the id) @param path: where the simcurves are written @param optfct: which optimisation function do you want me to use @param n: on how many sims do you want me to run the optimiser @param slist: ids of specific simcurves you want me to run on.

pycs.tdc.run.multirun(estimate, path, optfct, ncopy, nsim, clist=None, slist=None)¶

Wrapper around runsim and runobs Run the optimizer ncopy and nsim times on the copy and sim Return the results in a list of estimates objects

@param estimate: give me the estimate you want me to run on. (for the id) @param path: where the copycurves and simcurves are written @param optfct: which optimisation function do you want me to use @param ncopy: on how many copies do you want me to run the optimiser @param ncopy: on how many copies do you want me to run the optimiser @param slist: ids of specific copycurves you want me to run on. @param slist: ids of specific simcurves you want me to run on.

pycs.tdc.run.viz(estimate, path, datadir)¶: Look at some light curves. change in place…

pycs.tdc.run.summarize(estimate, path, makefig=False, skipdone=True)¶

Create an Estimate instance of a pair, that contains the results from the PyCS optimisation.

I will silently skip non-yet-ready pairs. If skipdone, I will silently skip those pairs that have already been summarized.

@param estimate: give me the estimate you want me to run on. (for the id) @param path: where the copycurves and simcurves are written @param makefig: do you want me to create a summary figure @param skipdone: do you want me to skip the pairs that are already summarized

pycs.tdc.run.collect(estimates, path)¶

Gather the output estimates created by summarize and return them in a single tuple

@param estimates: list of estimates you want me to collect @param path: where the summarized results are written

pycs.tdc.run2 module¶

pycs.tdc.run3 module¶

pycs.tdc.splopt module¶

Here we collect some spline optimizers specifically designed for the TDC

pycs.tdc.splopt.calcknotstep(varios)¶: Give me some outputs of vario, I try to return a good knotstep, based on the highest vratio between the curves and the sampling.

pycs.tdc.splopt.spl1(lcs, verbose=True)¶: Custom spline optimizer for TDC Assumes reasonable initial time shift, but no magshift.

pycs.tdc.splopt.spl2(lcs, maxit=7, minchange=1.0, verbose=True)¶

Custom spline optimizer for TDC Assumes good initial time shift, but does not care about inital magshift. Optimizes any ML that is present.

Parameters:	maxit – maximum number of iteartions minchange – minimum decrease in percent of the r2. I stop if the decrease gets smaller.

pycs.tdc.splopt.spl3(lcs, knotstepfact=1.0, mlknotstep=365, maxit=7, minchange=1.0, verbose=True)¶

Version 3 Assumes a really good initial time shift, does not care about inital magshift.

ML is added inside this function to a random lc. The ML signal should not get strong if it is not required.

Parameters:	maxit – maximum number of iteartions minchange – minimum decrease in percent of the r2. I stop if the decrease gets smaller.

pycs.tdc.splopt.splml1(lcs)¶: Some rather simple spline ML model. Randomly added on one curve of lcs

pycs.tdc.splopt.splml2(lcs)¶: Trying to get this better.

pycs.tdc.stats module¶

Compute some statistics about TDC1

pycs.tdc.stats.progress(estimates, htmlcode=False, path='leaderboard.html')¶

Print the overall progress of TDC1 estimations through D3CS

@param estimates: list of Estimate objects. @param htmlcode: boolean. If True, I will write the results in an html file @param path: path of the html output file.

pycs.tdc.util module¶

General purpose functions related to the TDC.

pycs.tdc.util.listtdc1v2pairs()¶

A list of all pairs that are available in TDC1 v2 (useful, as some pairs have been removed…)

@return: a list with all the existing pairs

pycs.tdc.util.tdcfilepath(set, rung, pair, skipset=False)¶

Return the path to a TDC light curve in the form of “set/rung/set_rung_pair.txt”

@param set: “tdc0” or “tdc1” @param rung: in [0, 4] for tdc1 @param pair: in [1, 1036] for tdc1 (some are missing) @param skipset: boolean. If true, skip the first “set” in the path

@return: string. Path “set/rung/set_rung_pair.txt”

pycs.tdc.util.pogmag(flux, fluxerr, m0=22.5)¶

Computes a “normal” Pogson magnitude from TDC info. Used for TDC0

@param flux: numpy array of floats @param fluxerr: numpy array of floats @param m0: float. Zero-point of the magnitude system. @return: tuple containing two lists of fluxes converted into magnitudes.

pycs.tdc.util.asinhmag(flux, fluxerr, m0=22.5, f0=1.0, b=0.01)¶

Implements asinh magnitudes, following http://ssg.astro.washington.edu/elsst/opsim.shtml?lightcurve_mags

@param flux: numpy array of floats @param fluxerr: numpy array of floats @param m0: float. Zero-point of the magnitude system. @param f0: float. asinh mag normalisation constant. @param b: float. Asinh mag normalisation constant. @return: tuple containing two lists of fluxes converted into magnitudes

pycs.tdc.util.read(filepath, mag='asinh', verbose=True, shortlabel=True)¶

Imports TDC light curves as a PyCS Lightcuve object. So far we expect exactly 2 curves in each file, as TDC simulates only doubles.

@param filepath: string. Path to the tdc lightcurves, typically the output of tdcfilepath @param mag: can be “asinh” of “pog”. Defines the type of magnitude considered @param verbose: boolean. Defines verbosity @param shortlabel: boolean. If True, gives a shorter object name to each light curve, omitting its TDC id.

@return: Return two PyCS Lightcurve objects, one per tdc light curve

pycs.tdc.util.writesubmission(estimates, filepath, commentlist=None, theseonly=False)¶

Writes a list of estimates into a TDC1 submission

@param estimates: list of Estimate objects, for which you want to write the time delay and error estimations. @param filepath: path of the file to be written. Accordign to the doc, filepath should be something like pycs_tdc1_algorithm.dt @param commentlist: list of srings. Additionnal comments to be written in the header of the submission file. @param theseonly: boolean. If set to True, I will not write -99 flags for those estimates that are missing in your list !

pycs.tdc.util.setnicemagshift(lcs)¶

Sets a “nice” magshift to the n-1 last curves of lcs, so that they appear nicely below each other when displayed. Also works if you curves contain ML models or are alredy shifted.

@param lcs: list of Lightcurve objects

pycs.tdc.util.goingon()¶: Asks the user if he wants to proceed. If not, exits python.

pycs.tdc.util.readsubmission(filepath)¶

Read a TDC1 submission file, and return its values in PyCS standards.

@param filepath: string. Path to the TDC1 submission

@return: list of tuples. Each tuple are (estimate id, estimated delay, estimated error)

pycs.tdc.util.godtweak(estimates)¶

I return a list of estimates in which I have modified some by divine inspiration. This should be systematically done before writing a TDC1-D3CS submission.

@param estimates: list of Estimate objects @return: tweaked list of estimates

pycs.tdc.vario module¶

Variability analysis stuff

pycs.tdc.vario.vario(l, plot=False, filepath=None, nsamp=1000000, verbose=False)¶

A simple stochatic variogram-like plot, and return the ratio delta_mag around 50-75 over delta_mag for the smallest step

@param l: light curve Object. @param nsamp: integer. The number of random samples to take @param plot: boolean. Do you want to plot the vario analysis ? @param filepath: if not None, it is the path to where the plot will be saved

Module contents¶

This subpackage contains code more or less specific to the Time Delay Challenge (tdc)