To be upright in heart is for your life to be facing toward the Lord as opposed to away from Him. When we face Him, we receive His blessings and see His glory and so we praise and thank Him and rejoice in Him.