Abstract In energy conservation research, energy disaggregation becomes an increasingly critical task, which takes a whole home electricity signal and decomposes it into its component appliances. While householder’s daily energy usage behavior acts as one powerful cue for breaking down the entire household’s energy consumption, existing works rarely modeled it straightforwardly. Instead, they either ignored the influence between users’ energy usage behaviors, or modeled the influence between the energy usages of appliances. With ambiguous appliance usage membership of householders, we find it difficult to appropriately model the influence between appliances, since such influence is determined by human behaviors in energy usage. To address this problem, we propose to model the influence between householders’ energy usage behaviors directly through a novel probabilistic model, which combines topic models with the Hawkes processes. The proposed model simultaneously disaggregates the whole home electricity signal into each component appliance and infers the appliance usage membership of household members, and enables those two tasks mutually benefit each other. Experimental results on both synthetic data and four real world data sets demonstrate the effectiveness of our model, which outperforms state-of-theart approaches in not only decomposing the entire consumed energy to each appliance in houses, but also the inference of household structures. We further analyze the inferred appliance-householder assignment and the corresponding influence within the appliance usage of each householder and across different householders, which provides insight into appealing human behavior patterns in appliance usage.