60种特征工程操作：使用自定义聚合函数

由 demi 提交于周五, 10 五月 2024 - 16:50

agg是一个聚合函数，使用指定轴上的一个或多个操作进行聚合。通过agg函数，可以同时对多列进行提取特征，非常适合用于特征工程。

<hr>

内置的聚合函数

在Pandas内部支持了13中聚合函数，可以在分组之后进行使用：

<ul><li>mean()：分组均值</li>
<li>sum()：分组求和</li>
<li>size()：分组个数</li>
<li>count()：分组大小</li>
<li>std()：分组标准差</li>
<li>var()：分组方差</li>
<li>sem()：均值误差</li>
<li>describe()：分组描述</li>
<li>first()：分组第一个元素</li>
<li>last()：分组最后一个元素</li>
<li>nth()：分组第N个元素</li>
<li>min()：分组最小值</li>
<li>max()：分组最大值</li></ul>

案例如下，有多种使用方式可供选择：

<pre># 定义模型
df = pd.DataFrame({'group':[1,1,2,2],
'values':[4,1,1,2],
'values2':[0,1,1,2]
})

# 分组对两列求均值
df.groupby('group').mean()

# 分组对两列求均值、标准差
df.groupby('group').agg([np.mean,np.std])

# 分组对两列分别聚合
df.groupby('group').agg(
{'values':['mean','median'],
'values2':['mean','std']}
)</pre>

<hr>

自定义聚合函数

如果在Pandas内部的聚合函数不满足要求，也可以自定义聚合函数搭配使用

▎median

<pre>def median(x):
return np.median(x)</pre>

▎variation_coefficient

<pre>def variation_coefficient(x):
mean = np.mean(x)
if mean != 0:
return np.std(x) / mean
else:
return np.nan</pre>

▎variance

<pre>def variance(x):
return np.var(x)</pre>

▎skewness

<pre>def skewness(x):
if not isinstance(x, pd.Series):
x = pd.Series(x)
return pd.Series.skew(x)</pre>

▎kurtosis

<pre>def kurtosis(x):
if not isinstance(x, pd.Series):
x = pd.Series(x)
return pd.Series.kurtosis(x)</pre>

▎standard_deviation

<pre>def standard_deviation(x):
return np.std(x)</pre>

▎large_standard_deviation

<pre>def large_standard_deviation(x):
if (np.max(x)-np.min(x)) == 0:
return np.nan
else:
return np.std(x)/(np.max(x)-np.min(x))</pre>

▎variation_coefficient

<pre>def variation_coefficient(x):
mean = np.mean(x)
if mean != 0:
return np.std(x) / mean
else:
return np.nan</pre>

▎variance_std_ratio

<pre>def variance_std_ratio(x):
y = np.var(x)
if y != 0:
return y/np.sqrt(y)
else:
return np.nan</pre>

▎ratio_beyond_r_sigma

<pre>def ratio_beyond_r_sigma(x, r):
if x.size == 0:
return np.nan
else:
return np.sum(np.abs(x - np.mean(x)) > r * np.asarray(np.std(x))) / x.size</pre>

▎range_ratio

<pre>def range_ratio(x):
mean_median_difference = np.abs(np.mean(x) - np.median(x))
max_min_difference = np.max(x) - np.min(x)
if max_min_difference == 0:
return np.nan
else:
return mean_median_difference / max_min_difference</pre>

▎has_duplicate_max

<pre>def has_duplicate_max(x):
return np.sum(x == np.max(x)) >= 2</pre>

▎has_duplicate_min

<pre>def has_duplicate_min(x):
return np.sum(x == np.min(x)) >= 2</pre>

▎has_duplicate

<pre>def has_duplicate(x):
return x.size != np.unique(x).size</pre>

▎count_duplicate_max

<pre>def count_duplicate_max(x):
return np.sum(x == np.max(x))</pre>

▎count_duplicate_min

<pre>def count_duplicate_min(x):
return np.sum(x == np.min(x))</pre>

▎count_duplicate

<pre>def count_duplicate(x):
return x.size - np.unique(x).size</pre>

▎sum_values

<pre>def sum_values(x):
if len(x) == 0:
return 0
return np.sum(x)</pre>

▎log_return

<pre>def log_return(list_stock_prices):
return np.log(list_stock_prices).diff() </pre>

▎realized_volatility

<pre>def realized_volatility(series):
return np.sqrt(np.sum(series**2))</pre>

▎realized_abs_skew

<pre>def realized_abs_skew(series):
return np.power(np.abs(np.sum(series**3)),1/3)</pre>

▎realized_skew

<pre>def realized_skew(series):
return np.sign(np.sum(series**3))*np.power(np.abs(np.sum(series**3)),1/3)</pre>

▎realized_vol_skew

<pre>def realized_vol_skew(series):
return np.power(np.abs(np.sum(series**6)),1/6)</pre>

▎realized_quarticity

<pre>def realized_quarticity(series):
return np.power(np.sum(series**4),1/4)</pre>

▎count_unique

<pre>def count_unique(series):
return len(np.unique(series))</pre>

▎count

<pre>def count(series):
return series.size</pre>

▎maximum_drawdown

<pre>def maximum_drawdown(series):
series = np.asarray(series)
if len(series)<2:
return 0
k = series[np.argmax(np.maximum.accumulate(series) - series)]
i = np.argmax(np.maximum.accumulate(series) - series)
if len(series[:i])<1:
return np.NaN
else:
j = np.max(series[:i])
return j-k</pre>

▎maximum_drawup

<pre>def maximum_drawup(series):
series = np.asarray(series)
if len(series)<2:
return 0

series = - series
k = series[np.argmax(np.maximum.accumulate(series) - series)]
i = np.argmax(np.maximum.accumulate(series) - series)
if len(series[:i])<1:
return np.NaN
else:
j = np.max(series[:i])
return j-k</pre>

▎drawdown_duration

<pre>def drawdown_duration(series):
series = np.asarray(series)
if len(series)<2:
return 0

k = np.argmax(np.maximum.accumulate(series) - series)
i = np.argmax(np.maximum.accumulate(series) - series)
if len(series[:i]) == 0:
j=k
else:
j = np.argmax(series[:i])
return k-j</pre>

▎drawup_duration

<pre>def drawup_duration(series):
series = np.asarray(series)
if len(series)<2:
return 0

series=-series
k = np.argmax(np.maximum.accumulate(series) - series)
i = np.argmax(np.maximum.accumulate(series) - series)
if len(series[:i]) == 0:
j=k
else:
j = np.argmax(series[:i])
return k-j</pre>

▎max_over_min

<pre>def max_over_min(series):
if len(series)<2:
return 0
if np.min(series) == 0:
return np.nan
return np.max(series)/np.min(series)</pre>

▎mean_n_absolute_max

<pre>def mean_n_absolute_max(x, number_of_maxima = 1):
""" Calculates the arithmetic mean of the n absolute maximum values of the time series."""
assert (
number_of_maxima > 0
), f" number_of_maxima={number_of_maxima} which is not greater than 1"

n_absolute_maximum_values = np.sort(np.absolute(x))[-number_of_maxima:]

return np.mean(n_absolute_maximum_values) if len(x) > number_of_maxima else np.NaN</pre>

▎count_above

<pre>def count_above(x, t):
if len(x)==0:
return np.nan
else:
return np.sum(x >= t) / len(x)</pre>

▎count_below

<pre>def count_below(x, t):
if len(x)==0:
return np.nan
else:
return np.sum(x <= t) / len(x)</pre>

▎number_peaks

<pre>def number_peaks(x, n):
x_reduced = x[n:-n]

res = None
for i in range(1, n + 1):
result_first = x_reduced > _roll(x, i)[n:-n]

if res is None:
res = result_first
else:
res &= result_first

res &= x_reduced > _roll(x, -i)[n:-n]
return np.sum(res)</pre>

▎mean_abs_change

<pre>def mean_abs_change(x):
return np.mean(np.abs(np.diff(x)))</pre>

▎mean_change

<pre>def mean_change(x):
x = np.asarray(x)
return (x[-1] - x[0]) / (len(x) - 1) if len(x) > 1 else np.NaN</pre>

▎mean_second_derivative_central

<pre>def mean_second_derivative_central(x):
x = np.asarray(x)
return (x[-1] - x[-2] - x[1] + x[0]) / (2 * (len(x) - 2)) if len(x) > 2 else np.NaN</pre>

▎root_mean_square

<pre>def root_mean_square(x):
return np.sqrt(np.mean(np.square(x))) if len(x) > 0 else np.NaN</pre>

▎absolute_sum_of_changes

<pre>def absolute_sum_of_changes(x):
return np.sum(np.abs(np.diff(x)))</pre>

▎longest_strike_below_mean

<pre>def longest_strike_below_mean(x):
if not isinstance(x, (np.ndarray, pd.Series)):
x = np.asarray(x)
return np.max(_get_length_sequences_where(x < np.mean(x))) if x.size > 0 else 0</pre>

▎longest_strike_above_mean

<pre>def longest_strike_above_mean(x):
if not isinstance(x, (np.ndarray, pd.Series)):
x = np.asarray(x)
return np.max(_get_length_sequences_where(x > np.mean(x))) if x.size > 0 else 0</pre>

▎count_above_mean

<pre>def count_above_mean(x):
m = np.mean(x)
return np.where(x > m)[0].size</pre>

▎count_below_mean

<pre>def count_below_mean(x):
m = np.mean(x)
return np.where(x < m)[0].size</pre>

▎last_location_of_maximum

<pre>def last_location_of_maximum(x):
x = np.asarray(x)
return 1.0 - np.argmax(x[::-1]) / len(x) if len(x) > 0 else np.NaN</pre>

▎first_location_of_maximum

<pre>def first_location_of_maximum(x):
if not isinstance(x, (np.ndarray, pd.Series)):
x = np.asarray(x)
return np.argmax(x) / len(x) if len(x) > 0 else np.NaN</pre>

▎last_location_of_minimum

<pre>def last_location_of_minimum(x):
x = np.asarray(x)
return 1.0 - np.argmin(x[::-1]) / len(x) if len(x) > 0 else np.NaN</pre>

▎first_location_of_minimum

<pre>def first_location_of_minimum(x):
if not isinstance(x, (np.ndarray, pd.Series)):
x = np.asarray(x)
return np.argmin(x) / len(x) if len(x) > 0 else np.NaN</pre>

▎percentage_of_reoccurring_values_to_all_values

<pre>def percentage_of_reoccurring_values_to_all_values(x):
if len(x) == 0:
return np.nan
unique, counts = np.unique(x, return_counts=True)
if counts.shape[0] == 0:
return 0
return np.sum(counts > 1) / float(counts.shape[0])</pre>

▎percentage_of_reoccurring_datapoints_to_all_datapoints

<pre>def percentage_of_reoccurring_datapoints_to_all_datapoints(x):
if len(x) == 0:
return np.nan
if not isinstance(x, pd.Series):
x = pd.Series(x)
value_counts = x.value_counts()
reoccuring_values = value_counts[value_counts > 1].sum()
if np.isnan(reoccuring_values):
return 0

return reoccuring_values / x.size</pre>

▎sum_of_reoccurring_values

<pre>def sum_of_reoccurring_values(x):
unique, counts = np.unique(x, return_counts=True)
counts[counts < 2] = 0
counts[counts > 1] = 1
return np.sum(counts * unique)</pre>

▎sum_of_reoccurring_data_points

<pre>def sum_of_reoccurring_data_points(x):
unique, counts = np.unique(x, return_counts=True)
counts[counts < 2] = 0
return np.sum(counts * unique)</pre>

▎ratio_value_number_to_time_series_length

<pre>def ratio_value_number_to_time_series_length(x):
if not isinstance(x, (np.ndarray, pd.Series)):
x = np.asarray(x)
if x.size == 0:
return np.nan

return np.unique(x).size / x.size</pre>

▎abs_energy

<pre>def abs_energy(x):
if not isinstance(x, (np.ndarray, pd.Series)):
x = np.asarray(x)
return np.dot(x, x)</pre>

▎quantile

<pre>def quantile(x, q):
if len(x) == 0:
return np.NaN
return np.quantile(x, q)</pre>

▎number_crossing_m

<pre>def number_crossing_m(x, m):
if not isinstance(x, (np.ndarray, pd.Series)):
x = np.asarray(x)
# From https://stackoverflow.com/questions/3843017/efficiently-detect-sign-cha…
positive = x > m
return np.where(np.diff(positive))[0].size</pre>

▎absolute_maximum

<pre>def absolute_maximum(x):
return np.max(np.absolute(x)) if len(x) > 0 else np.NaN</pre>

▎value_count

<pre>def value_count(x, value):
if not isinstance(x, (np.ndarray, pd.Series)):
x = np.asarray(x)
if np.isnan(value):
return np.isnan(x).sum()
else:
return x[x == value].size</pre>

▎range_count

<pre>def range_count(x, min, max):
return np.sum((x >= min) & (x < max))</pre>

▎mean_diff

<pre>def mean_diff(x):
return np.nanmean(np.diff(x.values))</pre>

本文转自：<a href="https://mp.weixin.qq.com/s/A2wV_CFkC_GBZ9VgpRfPZw">Coggle数据科学</a>，转载此文目的在于传递更多信息，版权归原作者所有。如不支持转载，请联系小编demi@eetrend.com删除。