Skip to content

Composite Functions

The eptr2.composite module provides pre-built functions that combine multiple API calls for common analysis tasks.

Overview

Composite functions simplify data retrieval by:

  • Combining multiple related API calls
  • Merging and transforming data
  • Handling missing data gracefully
  • Providing consistent output formats

Import

from eptr2.composite import (
    get_hourly_consumption_and_forecast_data,
    get_hourly_price_and_cost_data,
    get_hourly_production_data,
    get_hourly_production_plan_data,
    get_imbalance_data,
)

Function Reference

Consumption Functions

get_hourly_consumption_and_forecast_data 

get_hourly_consumption_and_forecast_data(start_date: str, end_date: str, eptr: EPTR2 | None = None, verbose: bool = False, include_contract_symbol: bool = False, **kwargs)

This composite function gets load plan, UECM (settlement consumption), real time and consumption data. If end date is after the last settlement data, UECM is filled with real time consumption under consumption column.

Source code in src/eptr2/composite/consumption.py 
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
def get_hourly_consumption_and_forecast_data(
    start_date: str,
    end_date: str,
    eptr: EPTR2 | None = None,
    verbose: bool = False,
    include_contract_symbol: bool = False,
    **kwargs,
):
    """
    This composite function gets load plan, UECM (settlement consumption), real time and consumption data. If end date is after the last settlement data, UECM is filled with real time consumption under consumption column.
    """

    if eptr is None:
        eptr = EPTR2(dotenv_path=kwargs.get("dotenv_path", ".env"))

    if verbose:
        print("Loading load plan...")

    lp_df = eptr.call(
        "load-plan",
        start_date=start_date,
        end_date=end_date,
        request_kwargs={"timeout": 5},
    )

    df = lp_df[["date", "lep"]].rename(columns={"lep": "load_plan", "date": "dt"})

    if verbose:
        print("Loading UECM...")

    uecm_df: pd.DataFrame = eptr.call(
        "uecm", start_date=start_date, end_date=end_date, request_kwargs={"timeout": 5}
    )

    if not uecm_df.empty:
        uecm_df = uecm_df[["period", "swv"]].rename(
            columns={"period": "dt", "swv": "uecm"}
        )

        df = df.merge(
            uecm_df,
            on="dt",
            how="outer",
        )

    if verbose:
        print("Loading real time consumption...")

    rt_cons = eptr.call(
        "rt-cons",
        start_date=start_date,
        end_date=end_date,
        request_kwargs={"timeout": 5},
    )

    df = df.merge(
        rt_cons[["date", "consumption"]].rename(
            columns={"date": "dt", "consumption": "rt_cons"}
        ),
        on="dt",
        how="outer",
    )

    if not uecm_df.empty:
        df["consumption"] = df.apply(
            lambda x: x["rt_cons"] if pd.isnull(x["uecm"]) else x["uecm"], axis=1
        )
    else:
        df["consumption"] = df["rt_cons"].copy()

    if include_contract_symbol:
        try:
            df["contract"] = df["dt"].apply(lambda x: iso_to_contract(x))
        except Exception as e:
            print("Contract information could not be added. Error:", e)

    return df

Price Functions

get_hourly_price_and_cost_data 

get_hourly_price_and_cost_data(start_date: str, end_date: str, eptr: EPTR2 | None = None, include_wap: bool = True, add_kupst_cost: bool = True, verbose: bool = False, include_contract_symbol: bool = True, timeout: int = 10, **kwargs)

This composite function gets price and imbalance (kupst included) cost data. Imbalance cost data consists of MCP (PTF), SMP (SMF), system direction (Enerji Açığı - up regulated, Energy Fazlası - down regulated, Dengede - Balanced), positive imbalance cost and negative imbalance cost.

Resulting columns are: - date: Datetime in ISO format and +03:00 timezone - mcp: Market Clearing Price (MCP/PTF) in TL (Turkish Lira) - smp: System Marginal Price (SMP/SMF) in TL (Turkish Lira) - wap: Weighted Average Price (WAP) in TL (Turkish Lira) - system_direction: System direction (Enerji Açığı - up regulated, Energy Fazlası - down regulated, Dengede - Balanced) - sd_sign: System direction sign (1 for Enerji Fazlası, -1 for Enerji Açığı, 0 for Dengede) - pos_imb_price: Positive imbalance price in TL (Turkish Lira) - neg_imb_price: Negative imbalance price in TL (Turkish Lira) (negative value) - pos_imb_cost: Positive imbalance cost in TL (Turkish Lira) (ptf - pos_imb_price) - neg_imb_cost: Negative imbalance cost in TL (Turkish Lira) (neg_imb_price - ptf)

Source code in src/eptr2/composite/price_and_cost.py 
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
def get_hourly_price_and_cost_data(
    start_date: str,
    end_date: str,
    eptr: EPTR2 | None = None,
    include_wap: bool = True,
    add_kupst_cost: bool = True,
    verbose: bool = False,
    include_contract_symbol: bool = True,
    timeout: int = 10,
    **kwargs,
):
    """
    This composite function gets price and imbalance (kupst included) cost data.
    Imbalance cost data consists of MCP (PTF), SMP (SMF), system direction (Enerji Açığı - up regulated, Energy Fazlası - down regulated, Dengede - Balanced), positive imbalance cost and negative imbalance cost.

    Resulting columns are:
    - date: Datetime in ISO format and +03:00 timezone
    - mcp: Market Clearing Price (MCP/PTF) in TL (Turkish Lira)
    - smp: System Marginal Price (SMP/SMF) in TL (Turkish Lira)
    - wap: Weighted Average Price (WAP) in TL (Turkish Lira)
    - system_direction: System direction (Enerji Açığı - up regulated, Energy Fazlası - down regulated, Dengede - Balanced)
    - sd_sign: System direction sign (1 for Enerji Fazlası, -1 for Enerji Açığı, 0 for Dengede)
    - pos_imb_price: Positive imbalance price in TL (Turkish Lira)
    - neg_imb_price: Negative imbalance price in TL (Turkish Lira) (negative value)
    - pos_imb_cost: Positive imbalance cost in TL (Turkish Lira) (ptf - pos_imb_price)
    - neg_imb_cost: Negative imbalance cost in TL (Turkish Lira) (neg_imb_price - ptf)
    """

    if eptr is None:
        eptr = EPTR2(dotenv_path=kwargs.get("dotenv_path", ".env"))

    if verbose:
        print("Getting MCP, SMP and imbalance price data...")

    max_lives = kwargs.get("max_lives", 2)

    lives = max_lives
    while lives > 0:
        try:
            price_df = eptr.call(
                "mcp-smp-imb",
                start_date=start_date,
                end_date=end_date,
                request_kwargs={"timeout": timeout},
            )
            break
        except Exception as e:
            lives -= 1
            if lives == 0:
                raise e
            else:
                if verbose:
                    print(
                        f"Error occurred while getting MCP, SMP and imbalance price data. Retrying... ({lives} attempts left)"
                    )

    price_df.drop(columns=["time"], inplace=True)

    price_df.rename(
        columns={
            "ptf": "mcp",
            "smf": "smp",
            "positiveImbalance": "pos_imb_price",
            "negativeImbalance": "neg_imb_price",
            "systemStatus": "system_direction",
        },
        inplace=True,
    )

    price_df["sd_sign"] = price_df["system_direction"].apply(
        lambda x: -1 if x == "Enerji Açığı" else 1 if x == "Enerji Fazlası" else 0
    )

    price_df["pos_imb_cost"] = price_df["mcp"] - price_df["pos_imb_price"]
    price_df["neg_imb_cost"] = price_df["neg_imb_price"] - price_df["mcp"]

    if include_wap:
        if verbose:
            print("Getting WAP data...")

        lives = max_lives
        while lives > 0:
            try:
                wap_df = eptr.call(
                    "wap",
                    start_date=start_date,
                    end_date=end_date,
                    request_kwargs={"timeout": timeout},
                )
                wap_df.drop(columns=["hour"], inplace=True)
                break
            except Exception as e:
                lives -= 1
                if lives == 0:
                    raise e
                else:
                    if verbose:
                        print(
                            f"Error occurred while getting WAP data. Retrying... ({lives} attempts left)"
                        )

        price_df = price_df.merge(wap_df, on="date", how="outer")

    if include_contract_symbol or add_kupst_cost:
        try:
            price_df["contract"] = price_df["date"].apply(lambda x: iso_to_contract(x))
        except Exception as e:
            print("Contract information could not be added. Error:", e)

    if add_kupst_cost:
        if verbose:
            print("Calculating unit KUPST cost...")

        added_params = {
            k: v
            for k, v in kwargs.items()
            if k
            in [
                "kupst_multiplier",
                "kupst_floor_price",
                "include_maintenance_penalty",
            ]
        }

        price_df["kupst_cost"] = price_df.apply(
            lambda x: calculate_unit_kupst_cost_by_contract(
                contract=x["contract"],
                mcp=x["mcp"],
                smp=x["smp"],
                **added_params,
            ),
            axis=1,
        )

        if not include_contract_symbol:
            price_df.drop(columns=["contract"], inplace=True)

    ### Column ordering
    columns_order = [
        "date",
        "contract",
        "pos_imb_vol",
        "neg_imb_vol",
        "pos_imb_mwh",
        "neg_imb_mwh",
        "mcp",
        "wap",
        "smp",
        "pos_imb_price",
        "neg_imb_price",
        "system_direction",
        "sd_sign",
        "pos_imb_cost",
        "neg_imb_cost",
        "kupst_cost",
    ]

    columns_order = [col for col in columns_order if col in price_df.columns]

    price_df = price_df[columns_order]

    return price_df

Production Functions

get_hourly_production_data 

get_hourly_production_data(start_date: str, end_date: str, eptr: EPTR2 | None = None, rt_pp_id: str | int | None = None, uevm_pp_id: str | int | None = None, verbose: bool = False, include_contract_symbol: bool = True, skip_uevm: bool = False, skip_rt: bool = False, **kwargs)

This composite function gets production data (Gerçek Zamanlı Üretim, UEVM) and merges them.

It is also possible to enter pp_id to get the production data for a specific production plan. Example ID values are given below.

rt_pp_id=641, ## ATATÜRK HES uevm_pp_id=142, ## ATATÜRK HES

Source code in src/eptr2/composite/production.py 
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
def get_hourly_production_data(
    start_date: str,
    end_date: str,
    eptr: EPTR2 | None = None,
    rt_pp_id: str | int | None = None,
    uevm_pp_id: str | int | None = None,
    verbose: bool = False,
    include_contract_symbol: bool = True,
    skip_uevm: bool = False,
    skip_rt: bool = False,
    **kwargs,
):
    """
    This composite function gets production data (Gerçek Zamanlı Üretim, UEVM) and merges them.

    It is also possible to enter pp_id to get the production data for a specific production plan. Example ID values are given below.

    rt_pp_id=641, ## ATATÜRK HES
    uevm_pp_id=142, ## ATATÜRK HES
    """

    if eptr is None:
        eptr = EPTR2(dotenv_path=kwargs.get("dotenv_path", ".env"))

    max_trials = kwargs.get("max_trials", 2)
    timeout = kwargs.get("timeout", 5)
    sleep_interval = kwargs.get("sleep_interval", 3)

    #### SANITY CHECKS ####
    if skip_rt and skip_uevm:
        raise ValueError("Both skip_rt and skip_uevm cannot be True.")

    #### REAL TIME GENERATION PHASE ####
    if skip_rt:
        rt_included = False
        rt_gen_df = pd.DataFrame()

    else:
        rt_included = True
        if verbose:
            print("Loading real time production data...")

        trials = max_trials
        while trials > 0:
            try:
                rt_gen_df: pd.DataFrame = eptr.call(
                    "rt-gen",
                    start_date=start_date,
                    end_date=end_date,
                    pp_id=rt_pp_id,
                    request_kwargs={"timeout": timeout},
                )
                break
            except Exception as e:
                trials -= 1

                if trials == 0:
                    raise e

                time.sleep(sleep_interval)
                if verbose:
                    print(
                        f"An error occurred while fetching RT data. Retrying after {sleep_interval} seconds... ({trials} attempts left)"
                    )

        if rt_gen_df.empty:
            raise ValueError("No data (production) is available for this date range.")

        try:
            rt_gen_df.drop("hour", axis=1, inplace=True)
        except Exception as e:
            if verbose:
                print(e)

        rt_gen_df.columns = [
            x + "_rt" if x not in ["date"] else x for x in rt_gen_df.columns
        ]

    #### UEVM PHASE ####
    if skip_uevm:
        uevm_included = False
        uevm_df = pd.DataFrame()

    else:
        uevm_included = True
        if verbose:
            print("Loading UEVM data...")

        trials = max_trials
        while trials > 0:
            try:
                uevm_df: pd.DataFrame = eptr.call(
                    "uevm",
                    start_date=start_date,
                    end_date=end_date,
                    pp_id=uevm_pp_id,
                    request_kwargs={"timeout": timeout},
                )
                break
            except Exception as e:
                trials -= 1
                if trials == 0:
                    raise e

                time.sleep(sleep_interval)
                if verbose:
                    print(
                        f"An error occurred while fetching UEVM data. Retrying after {sleep_interval} seconds... ({trials} attempts left)"
                    )

        try:
            uevm_df.drop("hour", axis=1, inplace=True)
        except Exception as e:
            within_settlement = check_date_for_settlement(x=end_date)
            if not within_settlement:
                print("Warning: The end date may not be within the settlement period.")
            if verbose:
                print(e)

        uevm_df.columns = [
            x + "_uevm" if x not in ["date"] else x for x in uevm_df.columns
        ]

    if uevm_included and rt_included:
        if verbose:
            print("Merging data...")

        if uevm_df.empty:
            merged_df = rt_gen_df.copy()
        else:
            merged_df = rt_gen_df.merge(uevm_df, how="left", on=["date"])
    elif uevm_included:
        merged_df = uevm_df.copy()
    elif rt_included:
        merged_df = rt_gen_df.copy()

    if include_contract_symbol:
        try:
            merged_df["contract"] = merged_df["date"].apply(
                lambda x: iso_to_contract(x)
            )
        except Exception as e:
            print("Contract information could not be added. Error:", e)

    merged_df = merged_df.rename(columns={"date": "dt"})

    return merged_df

Quick Reference

get_hourly_consumption_and_forecast_data

Combines load plan, UECM, and real-time consumption data.

from eptr2.composite import get_hourly_consumption_and_forecast_data

df = get_hourly_consumption_and_forecast_data(
    start_date="2024-07-01",
    end_date="2024-07-31",
    verbose=True
)

Returns:

Column Type Description
dt datetime Timestamp
load_plan float Load plan forecast (MWh)
uecm float Settlement consumption (MWh)
rt_cons float Real-time consumption (MWh)
consumption float Best available consumption

get_hourly_price_and_cost_data

Combines MCP, SMP, and imbalance price data.

from eptr2.composite import get_hourly_price_and_cost_data

df = get_hourly_price_and_cost_data(
    start_date="2024-07-01",
    end_date="2024-07-31"
)

Returns:

Column Type Description
dt datetime Timestamp
mcp float Market Clearing Price (TL/MWh)
smp float System Marginal Price (TL/MWh)
positive_imbalance float Positive imbalance price
negative_imbalance float Negative imbalance price

get_hourly_production_data

Retrieves hourly production data by source.

from eptr2.composite import get_hourly_production_data

df = get_hourly_production_data(
    start_date="2024-07-01",
    end_date="2024-07-31"
)

get_imbalance_data

Retrieves imbalance-related data for cost calculations.

from eptr2.composite import get_imbalance_data

df = get_imbalance_data(
    start_date="2024-07-01",
    end_date="2024-07-31"
)

Common Parameters

All composite functions share these parameters:

Parameter Type Default Description
start_date str Required Start date (YYYY-MM-DD)
end_date str Required End date (YYYY-MM-DD)
eptr EPTR2 None EPTR2 instance (created if not provided)
verbose bool False Print progress messages
**kwargs - - Additional parameters

Examples

Complete Analysis Pipeline

from eptr2 import EPTR2
from eptr2.composite import (
    get_hourly_consumption_and_forecast_data,
    get_hourly_price_and_cost_data
)
import pandas as pd

# Initialize once
eptr = EPTR2(use_dotenv=True, recycle_tgt=True)

# Get consumption data
consumption = get_hourly_consumption_and_forecast_data(
    start_date="2024-07-01",
    end_date="2024-07-31",
    eptr=eptr,
    verbose=True
)

# Get price data
prices = get_hourly_price_and_cost_data(
    start_date="2024-07-01",
    end_date="2024-07-31",
    eptr=eptr
)

# Merge for analysis
analysis = pd.merge(consumption, prices, on='dt')

# Calculate costs
analysis['total_cost'] = analysis['consumption'] * analysis['mcp']
print(f"Total cost: {analysis['total_cost'].sum():,.2f} TL")

With Verbose Output

df = get_hourly_consumption_and_forecast_data(
    start_date="2024-07-01",
    end_date="2024-07-31",
    verbose=True
)

Output: 

Loading load plan...
Loading UECM...
Loading real time consumption...

Using Custom EPTR2 Instance

from eptr2 import EPTR2
from eptr2.composite import get_hourly_consumption_and_forecast_data

# Custom configuration
eptr = EPTR2(
    use_dotenv=True,
    recycle_tgt=True,
    tgt_path="/custom/path"
)

df = get_hourly_consumption_and_forecast_data(
    start_date="2024-07-01",
    end_date="2024-07-31",
    eptr=eptr
)

See Also