James is talking about unfaithfulness to the Lord.

Since our Lord is SpiritSoulAndBody it is possible to be unfaithful -> to His Spirit -> to His Soul and -> to His Body

A person is unfaithful to the Holy SpiritOfChrist Jesus by entertaining evil spirits.

A person is unfaithful to the Soul of Christ by denying His Word.

A person is unfaithful to the Body of Christ by adding works to Grace.