hybridvut

This package provides functionalities to deal with hybrid IO-LCA based on the Make and Use framework.

Key features are:

• hybridization of a foreground and a background system
• unit handling using iam_units (pint units)
• functions for data handling (e.g. region aggregation, RAS)
• basic IO calculations (e.g. transaction and multiplier matrices)
• preprocessing to bring raw data into the right format (i.e. message-ix data)

Additional hybridization procedures might be implemented in future work. Furthermore, functions can be also applied just to a single system.

Basic structure

The general idea is to distinguish between a foreground and a background system. The forground system and the background system are represented by a make table (V_for and V_back, respectively) and a use table (U_for and U_back , respectively).

The aim is to combine both systems so that no double-counting occurs. Special emphasize is thus needed for the use table U.

To connect both systems properly, co-called cut-off matrices C_u and C_d are introduced. C_u connects the upstream processes of the foreground system to the industries of the background system. C_d connects the downstream industries to the processes of the foreground sytem. V_for, V_back, U_for and U_back are known.

The hybridized total system represented by V and U includes all necessary adoptions of the submatrices and avoids double-counting.

-	Make Table V	----------->	Use Table U	---------->
indices	cf	cb	if	ib
cf			Ufor	Cd
cb			Cu	Uback
if	Vfor	0
ib	0	Vback

Furthermore, additional factors can be taken into account which are represented by the intervention matrix F.

To re-allocate the interventions from the industries of the background system to the foreground system, a sub-matrix F_u is introduced.

NOTE: An additional sub-matrix is thinkable to re-allocate interventions the other way around, but is not yet implemented. The intervention tables might be in principle also defined per commodity.

-	Intervention Table F	------------------>	Intervention Table F	------------------>
indices	cf	cb	if	ib
kf	0	0	Ffor	0
kb	0	0	Fu	Fback

Data format

The hybridvut package uses pandas.DataFrame to deal with the required matrices. The format of these DataFrames needs to fulfill certain requirements in regard to its pandas.MultiIndex.

Make matrix V:

-	region	...	...	...
	commodity	...	...	...
	unit	...	...	...
region	industry
...	...	v1,1	v1,2	...
...	...	v2,1	v2,2	...
...	...	...	...	...

Use matrix U:

-	-	region	...	...	...
		industry	...	...	...
region	commodity	unit
...	...	...	u1,1	u1,2	...
...	...	...	u2,1	u2,2	...
...	...	...	...	...	...

Intervention matrix F:

-	region	...	...	...
	industry	...	...	...
intervention	unit
...	...	f1,1	f1,2	...
...	...	f2,1	f2,2	...
...	...	...	...	...

Characterization matrix Q:

-	intervention	...	...	...
	unit	...	...	...
impact	unit
...	...	q1,1	q1,2	...
...	...	q2,1	q2,2	...
...	...	...	...	...

Hybridization procedure

This section describes the current core procedure for hybridization.

import hybridvut as hyb

foreground = hyb.VUT(V=V_for, U=U_for, F=F_for, Q=None)
background = hyb.VUT(V=V_back, U=U_back, F=F_back, Q=Q_back)

HT = hyb.HybridTables(forground=foreground, background=background)
HT.hybridize(H, H1, HF)

# Hybridized total system
HT.total.V
HT.total.U
HT.total.F
HT.total.Q

1. Define concordance matrices H, H1 and HF (for industries, commodities and interventions, respectively)
2. Adjust V_back, U_back; and calculate q and x based on V
3. Define weighting factor matrices T and T1
4. Calculate upstream cut-off matrix C_u and adjust U_back (second time)
5. Calculate downstream cut-off matrix C_d and adjust U_back (third time)
6. Correct C_d, C_u and U_for
7. Adjust intervention sub-tables of F
8. Combine all submatrices into V, U and F
9. Adjust characterization matrix Q

1) Define concordance matrices H ($H_{ind}$), H1 ($H_{com}$) and HF ($H_{int}$)

$H_{ind}$ relates process of foreground system to industry of background system (includes 1 if relation, otherwise 0).

$H_{com}$ relates commodity of foreground system to commodity of background system (includes 1 if relation, otherwise 0).

$H_{int}$ relates an intervention of the foreground system to an intervention of the background system (includes 1 if relation, otherwise 0).

Note: in multi-regional models the relation is country-wise (e.g., a commodity of a country can only related to a commodity of one country). However, more than one industry of the background can be associated to the industry of the forground system (the same is true for the commodity relation). Thereby, the sum of all associated industries (commodities) must equal 1.

Concordance matrix H for industries:

-	region	...	...	...
	industry	...	...	...
region	industry
...	...	h1b,1f	h1b,2f	...
...	...	h2b,1f	h2b,2f	...
...	...	...	...	...

Concordance matrix H1 for commodities:

-	-	region	...	...	...
		commodity	...	...	...
		unit	...	...	...
region	commodity	unit
...	...	...	h11f,1b	h11f,2b	...
...	...	...	h12f,1b	h12f,2b	...
...	...	...	...	...	...

Concordance matrix HF for interventions:

-	intervention	...	...	...
	unit	...	...	...
intervention	unit
...	...	hf1b,1f	hf1b,2f	...
...	...	hf2b,1f	hf2b,2f	...
...	...	...	...	...

2 Adjust $V_{back}$, $U_{back}$; and calculate $q$ and $g$ based on $V$

\mathbf{U}_{back}^{adj} = \mathbf{U}_{back} - \mathbf{H}_{com}^{T} \cdot \mathbf{U}_{for} \cdot \mathbf{H}_{ind}^{T} \\
\mathbf{V}_{back}^{adj} = \mathbf{V}_{back} - \mathbf{H}_{ind} \cdot \mathbf{V}_{for} \cdot \mathbf{H}_{com} \\
\mathbf{g}_{for} = \mathbf{V}_{for} \cdot \mathbf{i} \\
\mathbf{q}_{for} = \mathbf{V}_{for}^{T} \cdot \mathbf{i} \\
\mathbf{g}_{back}^{adj} = \mathbf{V}_{back}^{adj} \cdot \mathbf{i} \\
\mathbf{q}_{back}^{adj} = \mathbf{V}_{back}^{adj T} \cdot \mathbf{i} 

, where $ì$ is the summation vector of ones with prpoper lenght.

3) Calculate weighting factor matrices T ($T_{u}$) and T1 ($T_{d}$)

Matrix $T_{u}$ defines how much of the upstream input can be related to the foreground processes (and thus shifted from the background industries). Here, the assumption is made that this is determined automatically by the ratio of industry output x (of the new process and its reference industry).

\mathbf{T}_{u} = \left[ \mathbf{J}_{c,i} \cdot \mathbf{H}_{ind} \cdot \mathbf{\hat{g}_{for}} \right]  \varnothing  \left[ \mathbf{J}_{c,i} \cdot \mathbf{H}_{ind} \cdot \mathbf{\hat{g}_{for}} + \mathbf{J}_{c,i} \cdot \mathbf{\hat{x}_{back}^{adj}} \cdot \mathbf{H}_{ind}  \right]

Matrix$T_{d}$ defines how much of the downstream input can be related to the foreground processes (and thus shifted from the background industries). Here, the assumption is made that this is determined by the ratio of commodity output q (of the new commodity and its reference commodity).

\mathbf{T}_{d} = \left[  \mathbf{\hat{q}_{for}}  \cdot \mathbf{H}_{com} \cdot  \mathbf{J}_{c,i}  \right]  \varnothing  \left[ \mathbf{\hat{q}_{for}}  \cdot \mathbf{H}_{com} \cdot  \mathbf{J}_{c,i} + \mathbf{H}_{com} \cdot \mathbf{\hat{q}_{back}^{adj}}  \cdot \mathbf{J}_{c,i} \right]

4) Calculate upstream cut-off matrix $C_{u}$ and adjust $U_{back}$ (second time)

\mathbf{C}_{u} = \mathbf{T}_{u} \odot \mathbf{U}_{back}^{adj} \cdot \mathbf{H}_{ind} \\
\mathbf{U}_{back}^{adj1} = \mathbf{U}_{back}^{adj} - \mathbf{C}_{u} \cdot \mathbf{H}_{ind}^{T} 

5) Calculate downstream cut-off matrix $C_{d}$ and adjust $U_{back}$ (third time)

\mathbf{C}_{d} = \mathbf{T}_{d} \odot \mathbf{H}_{com} \cdot \mathbf{U}_{back}^{adj1} \\
\mathbf{U}_{back}^{adj2} = \mathbf{U}_{back}^{adj1} - \mathbf{H}_{com}^{T} \cdot \mathbf{C}_{d} 

6) Correct $C_{u}$, $C_{d}$ and $U_{for}$

The correction term $\mathbf{O}_{u}$ removes commodities those are actually covered as commodities in the foreground system. The correction term $\mathbf{O}_{d}$ removes commodities those are actually used by technology of the foreground system.

\mathbf{C}_{u}^{adj} = \mathbf{C}_{u} - \underbrace{ \mathbf{C}_{u} \odot \mathbf{H}_{com}^{T} \cdot \mathbf{T}_{u} \cdot \mathbf{H}_{ind} }_{\text{correction term $\mathbf{O}_{u}$}}\\
\mathbf{C}_{d}^{adj} = \mathbf{C}_{d} - \underbrace{ \mathbf{C}_{d} \odot \mathbf{H}_{com} \cdot \mathbf{T}_{d} \cdot \mathbf{H}_{ind}^{T}}_{\text{correction term $\mathbf{O}_{d}$}} \\
\mathbf{U}_{for}^{adj} = \mathbf{U}_{for} + \mathbf{H}_{com} \cdot \mathbf{O}_{u} + \mathbf{O}_{d} \cdot \mathbf{H}_{ind}

7) Adjust intervention sub-tables of $F$

The re-allocation procedure is similar to the first steps of re-allocation the commodities from the background to the foreground system. The implicit assumption is (currently) that the background system already includes the intire interventions; and no additional interventions are added by the foreground system. The task hence is to remove the interventions of the associated processes from the background system to the foreground system, which are not yet taken into account in the foreground system.

\mathbf{F}_{back}^{adj} = \mathbf{F}_{back} - \mathbf{H}_{inter} \cdot \mathbf{F}_{for} \cdot \mathbf{H}_{ind}^{T} \\
\mathbf{T}_{inter} = \left[ \mathbf{J}_{c,i} \cdot \mathbf{H}_{ind} \cdot \mathbf{\hat{g}_{for}} \right]  \varnothing  \left[ \mathbf{J}_{c,i} \cdot \mathbf{H}_{ind} \cdot \mathbf{\hat{g}_{for}} + \mathbf{J}_{c,i} \cdot \mathbf{\hat{g}_{back}^{adj}} \cdot \mathbf{H}_{ind}  \right] \\
\mathbf{F}_{u} = \mathbf{T}_{inter} \odot \mathbf{F}_{back}^{adj} \cdot \mathbf{H}_{ind} \\ 
\mathbf{F}_{back}^{adj1} = \mathbf{F}_{back}^{adj} - \mathbf{F}_{u} \cdot \mathbf{H}_{ind}^{T}

8) Combine all submatrices into $V$, $U$ and $F$

-	Make Table V	----------->	Use Table U	---------->
indices	cf	cb	if	ib
cf			Uforadj	Cdadj
cb			Cuadj	Ubackadj2
if	Vfor	0
ib	0	Vbackadj

-	Intervention Table F	--------------------->	Intervention Table F	------------------->
indices	cf	cb	if	ib
kf	0	0	Ffor	0
kb	0	0	Fu	Fbackadj1

9) Adjust characterization matrix $Q$

To deal with the characterization factors of (environmental) interventions, currently, two alternatives are implemented:

• Characterisation factors are provided for the background system; or
• Characterization factors are provided for the foreground system

If Q_back is given:

\mathbf{Q}_{for} = \mathbf{Q}_{back} \cdot \mathbf{H}_{int}

If Q_for is given:

\mathbf{Q}_{back} = \mathbf{Q}_{for} \cdot \mathbf{H}_{int}^{T}

-	Characterization Table Q	--------------------->
indices	kf	kb
of or b	Qfor	Qback

Further notes

The hybridization procedure for the messageix-exiobase example includes three modifications of these steps (necessary due to the nature of the messageix model):

1. Aggregating sectors (i.e., technologies those have "final" products as input flows) are set to zero. The use of these products is covered by the background system.
2. To ensure that value added is the same after hybridizarion, we use it from the background system ($F_{back}$), allocate it among the industries (i.e. foreground technologies) and resacle the make and use tables to fulfill the original Exiobase ratios of values added ($v_{back}_{i}$) per intermediate input ($\sum_{c} U_{back}_{c,i}$) of the corresponding industry i.
3. Before allocating GHG emissions from background to foreground industries (step 7), the total GHG emissions (CO2, CH4, N20), are usd to scale the corresponding emissions in Exiobase. GHG emissions of the foreground system are then set to zero ($F_{for}$).